2023-12-02T02:33:07.369 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-02T02:33:07.369 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-12-01_18:01:25-rbd-main-distro-default-smithi/7474803 --name yuriw-2023-12-01_18:01:25-rbd-main-distro-default-smithi --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} --owner scheduled_yuriw@teuthology --verbose -- /home/teuthworker/archive/yuriw-2023-12-01_18:01:25-rbd-main-distro-default-smithi/7474803/orig.config.yaml 2023-12-02T02:33:07.417 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-02T02:33:07.507 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-01_18:01:25-rbd-main-distro-default-smithi/7474803 branch: main description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} email: yweinste@redhat.com first_in_suite: false job_id: '7474803' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: yuriw-2023-12-01_18:01:25-rbd-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: simple client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: bed7681480ccb0192b91a35e691658867e1e96b7 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: bed7681480ccb0192b91a35e691658867e1e96b7 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: bed7681480ccb0192b91a35e691658867e1e96b7 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 2451 sha1: bed7681480ccb0192b91a35e691658867e1e96b7 sleep_before_teardown: 0 subset: 111/120000 suite: rbd suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: bed7681480ccb0192b91a35e691658867e1e96b7 targets: smithi029.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHhvEccHXaj9KecRdhJ2R7k6deK/FnDjkxOTwaL94Ze0GkNEzZbozrPwNc2SMmJZq3gdDq6vTnghyy+IyRAxjQw= smithi057.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBcE1vhLAy0ZftcWRW+Iu01zQaKILaNE+BFn/TIqb3zF05SMgiwusZ/3kJVVvXngHrx6SL2rEFn+kkOfgIxp3HY= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_MODE: snapshot RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-01_18:01:25 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-02T02:33:07.507 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa; will attempt to use it 2023-12-02T02:33:07.508 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks 2023-12-02T02:33:07.508 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-02T02:33:07.510 INFO:teuthology.task.internal:Checking packages... 2023-12-02T02:33:07.533 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'bed7681480ccb0192b91a35e691658867e1e96b7' 2023-12-02T02:33:07.534 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-02T02:33:07.534 INFO:teuthology.packaging:ref: None 2023-12-02T02:33:07.534 INFO:teuthology.packaging:tag: None 2023-12-02T02:33:07.534 INFO:teuthology.packaging:branch: main 2023-12-02T02:33:07.534 INFO:teuthology.packaging:sha1: bed7681480ccb0192b91a35e691658867e1e96b7 2023-12-02T02:33:07.534 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2023-12-02T02:33:07.780 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7660.gb27938f5 2023-12-02T02:33:07.780 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-02T02:33:07.786 INFO:teuthology.task.internal:no buildpackages task found 2023-12-02T02:33:07.786 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-02T02:33:07.817 INFO:teuthology.task.internal:Saving configuration 2023-12-02T02:33:07.836 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-02T02:33:07.854 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-02T02:33:07.879 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-01_18:01:25-rbd-main-distro-default-smithi/7474803', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-02 02:25:59.240375', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHhvEccHXaj9KecRdhJ2R7k6deK/FnDjkxOTwaL94Ze0GkNEzZbozrPwNc2SMmJZq3gdDq6vTnghyy+IyRAxjQw='} 2023-12-02T02:33:07.900 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi057.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-01_18:01:25-rbd-main-distro-default-smithi/7474803', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-02 02:25:59.242125', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBcE1vhLAy0ZftcWRW+Iu01zQaKILaNE+BFn/TIqb3zF05SMgiwusZ/3kJVVvXngHrx6SL2rEFn+kkOfgIxp3HY='} 2023-12-02T02:33:07.900 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-02T02:33:07.906 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2023-12-02T02:33:07.906 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2023-12-02T02:33:07.907 INFO:teuthology.run_tasks:Running task console_log... 2023-12-02T02:33:08.001 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f985f90f0d0>, signals=[15]) 2023-12-02T02:33:08.001 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-02T02:33:08.007 INFO:teuthology.task.internal:Opening connections... 2023-12-02T02:33:08.007 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com 2023-12-02T02:33:08.009 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-02T02:33:08.109 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi057.front.sepia.ceph.com 2023-12-02T02:33:08.110 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-02T02:33:08.208 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-02T02:33:08.214 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2023-12-02T02:33:08.232 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2023-12-02T02:33:08.232 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2023-12-02T02:33:08.288 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Stream" 2023-12-02T02:33:08.288 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="9" 2023-12-02T02:33:08.288 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos" 2023-12-02T02:33:08.288 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora" 2023-12-02T02:33:08.289 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="9" 2023-12-02T02:33:08.289 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el9" 2023-12-02T02:33:08.289 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-02T02:33:08.289 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31" 2023-12-02T02:33:08.289 INFO:teuthology.orchestra.run.smithi029.stdout:LOGO="fedora-logo-icon" 2023-12-02T02:33:08.289 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-02T02:33:08.289 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/" 2023-12-02T02:33:08.289 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-02T02:33:08.289 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-02T02:33:08.289 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-02T02:33:08.290 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2023-12-02T02:33:08.313 DEBUG:teuthology.orchestra.run.smithi057:> uname -m 2023-12-02T02:33:08.330 INFO:teuthology.orchestra.run.smithi057.stdout:x86_64 2023-12-02T02:33:08.331 DEBUG:teuthology.orchestra.run.smithi057:> cat /etc/os-release 2023-12-02T02:33:08.388 INFO:teuthology.orchestra.run.smithi057.stdout:NAME="CentOS Stream" 2023-12-02T02:33:08.388 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION="9" 2023-12-02T02:33:08.388 INFO:teuthology.orchestra.run.smithi057.stdout:ID="centos" 2023-12-02T02:33:08.388 INFO:teuthology.orchestra.run.smithi057.stdout:ID_LIKE="rhel fedora" 2023-12-02T02:33:08.388 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION_ID="9" 2023-12-02T02:33:08.388 INFO:teuthology.orchestra.run.smithi057.stdout:PLATFORM_ID="platform:el9" 2023-12-02T02:33:08.388 INFO:teuthology.orchestra.run.smithi057.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-02T02:33:08.388 INFO:teuthology.orchestra.run.smithi057.stdout:ANSI_COLOR="0;31" 2023-12-02T02:33:08.388 INFO:teuthology.orchestra.run.smithi057.stdout:LOGO="fedora-logo-icon" 2023-12-02T02:33:08.388 INFO:teuthology.orchestra.run.smithi057.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-02T02:33:08.388 INFO:teuthology.orchestra.run.smithi057.stdout:HOME_URL="https://centos.org/" 2023-12-02T02:33:08.389 INFO:teuthology.orchestra.run.smithi057.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-02T02:33:08.389 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-02T02:33:08.389 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-02T02:33:08.389 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server 2023-12-02T02:33:08.412 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-02T02:33:08.422 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-02T02:33:08.455 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-02T02:33:08.456 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest 2023-12-02T02:33:08.459 DEBUG:teuthology.orchestra.run.smithi057:> test '!' -e /home/ubuntu/cephtest 2023-12-02T02:33:08.474 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-02T02:33:08.485 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-02T02:33:08.485 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph) 2023-12-02T02:33:08.515 DEBUG:teuthology.orchestra.run.smithi057:> test -z $(ls -A /var/lib/ceph) 2023-12-02T02:33:08.541 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-02T02:33:08.595 INFO:teuthology.run_tasks:Running task kernel... 2023-12-02T02:33:08.609 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-02T02:33:08.610 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-02T02:33:08.610 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-02T02:33:08.610 DEBUG:teuthology.orchestra.run.smithi057:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-02T02:33:08.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-02T02:33:08.627 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2023-12-02T02:33:08.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-02T02:33:08.628 DEBUG:teuthology.orchestra.run.smithi057:> uname -r 2023-12-02T02:33:08.683 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-361.el9.x86_64 2023-12-02T02:33:08.684 INFO:teuthology.task.kernel:Running kernel on smithi029: 5.14.0-361.el9.x86_64 2023-12-02T02:33:08.684 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum install -y kernel 2023-12-02T02:33:08.686 INFO:teuthology.orchestra.run.smithi057.stdout:5.14.0-361.el9.x86_64 2023-12-02T02:33:08.686 INFO:teuthology.task.kernel:Running kernel on smithi057: 5.14.0-361.el9.x86_64 2023-12-02T02:33:08.686 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum install -y kernel 2023-12-02T02:33:10.753 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS-9 - AppStream 14 MB/s | 6.4 MB 00:00 2023-12-02T02:33:11.857 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - AppStream 12 MB/s | 6.4 MB 00:00 2023-12-02T02:33:12.179 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2023-12-02T02:33:13.078 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 7.9 MB 00:00 2023-12-02T02:33:13.316 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - BaseOS 5.8 MB/s | 2.0 MB 00:00 2023-12-02T02:33:15.079 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - AppStream 22 MB/s | 18 MB 00:00 2023-12-02T02:33:15.125 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 5.4 MB/s | 7.9 MB 00:01 2023-12-02T02:33:19.012 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 6.6 MB/s | 18 MB 00:02 2023-12-02T02:33:19.113 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - CRB 12 MB/s | 5.8 MB 00:00 2023-12-02T02:33:20.763 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 15 kB 00:00 2023-12-02T02:33:22.376 INFO:teuthology.orchestra.run.smithi057.stdout:Extra Packages for Enterprise Linux 13 MB/s | 20 MB 00:01 2023-12-02T02:33:23.730 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 4.8 MB/s | 5.8 MB 00:01 2023-12-02T02:33:25.374 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 15 kB 00:00 2023-12-02T02:33:26.728 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 16 MB/s | 20 MB 00:01 2023-12-02T02:33:27.331 INFO:teuthology.orchestra.run.smithi057.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2023-12-02T02:33:29.779 INFO:teuthology.orchestra.run.smithi057.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-02T02:33:29.779 INFO:teuthology.orchestra.run.smithi057.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-02T02:33:29.877 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-12-02T02:33:29.883 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:33:29.883 INFO:teuthology.orchestra.run.smithi057.stdout: Package Architecture Version Repository Size 2023-12-02T02:33:29.883 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:33:29.883 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2023-12-02T02:33:29.883 INFO:teuthology.orchestra.run.smithi057.stdout: kernel x86_64 5.14.0-388.el9 baseos 5.6 M 2023-12-02T02:33:29.883 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2023-12-02T02:33:29.883 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-core x86_64 5.14.0-388.el9 baseos 20 M 2023-12-02T02:33:29.883 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-modules x86_64 5.14.0-388.el9 baseos 38 M 2023-12-02T02:33:29.884 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-modules-core x86_64 5.14.0-388.el9 baseos 32 M 2023-12-02T02:33:29.884 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:33:29.884 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2023-12-02T02:33:29.884 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:33:29.884 INFO:teuthology.orchestra.run.smithi057.stdout:Install 4 Packages 2023-12-02T02:33:29.884 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:33:29.884 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 96 M 2023-12-02T02:33:29.885 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 121 M 2023-12-02T02:33:29.885 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2023-12-02T02:33:31.233 INFO:teuthology.orchestra.run.smithi057.stdout:(1/4): kernel-5.14.0-388.el9.x86_64.rpm 4.3 MB/s | 5.6 MB 00:01 2023-12-02T02:33:31.700 INFO:teuthology.orchestra.run.smithi057.stdout:(2/4): kernel-core-5.14.0-388.el9.x86_64.rpm 11 MB/s | 20 MB 00:01 2023-12-02T02:33:31.729 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2023-12-02T02:33:32.617 INFO:teuthology.orchestra.run.smithi057.stdout:(3/4): kernel-modules-5.14.0-388.el9.x86_64.rpm 14 MB/s | 38 MB 00:02 2023-12-02T02:33:33.816 INFO:teuthology.orchestra.run.smithi057.stdout:(4/4): kernel-modules-core-5.14.0-388.el9.x86_6 13 MB/s | 32 MB 00:02 2023-12-02T02:33:33.817 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:33:33.817 INFO:teuthology.orchestra.run.smithi057.stdout:Total 24 MB/s | 96 MB 00:03 2023-12-02T02:33:34.186 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-02T02:33:34.187 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-02T02:33:34.293 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-02T02:33:34.309 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:33:34.309 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2023-12-02T02:33:34.309 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:33:34.309 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-02T02:33:34.309 INFO:teuthology.orchestra.run.smithi029.stdout: kernel x86_64 5.14.0-388.el9 baseos 5.6 M 2023-12-02T02:33:34.309 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-12-02T02:33:34.309 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core x86_64 5.14.0-388.el9 baseos 20 M 2023-12-02T02:33:34.309 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules x86_64 5.14.0-388.el9 baseos 38 M 2023-12-02T02:33:34.310 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core x86_64 5.14.0-388.el9 baseos 32 M 2023-12-02T02:33:34.310 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:33:34.310 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-02T02:33:34.310 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:33:34.310 INFO:teuthology.orchestra.run.smithi029.stdout:Install 4 Packages 2023-12-02T02:33:34.310 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:33:34.311 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 96 M 2023-12-02T02:33:34.311 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 121 M 2023-12-02T02:33:34.311 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-02T02:33:34.705 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2023-12-02T02:33:34.732 INFO:teuthology.orchestra.run.smithi029.stdout:(1/4): kernel-5.14.0-388.el9.x86_64.rpm 18 MB/s | 5.6 MB 00:00 2023-12-02T02:33:35.136 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2023-12-02T02:33:35.136 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2023-12-02T02:33:35.207 INFO:teuthology.orchestra.run.smithi029.stdout:(2/4): kernel-core-5.14.0-388.el9.x86_64.rpm 26 MB/s | 20 MB 00:00 2023-12-02T02:33:35.999 INFO:teuthology.orchestra.run.smithi029.stdout:(3/4): kernel-modules-5.14.0-388.el9.x86_64.rpm 24 MB/s | 38 MB 00:01 2023-12-02T02:33:36.499 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2023-12-02T02:33:36.499 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2023-12-02T02:33:36.525 INFO:teuthology.orchestra.run.smithi029.stdout:(4/4): kernel-modules-core-5.14.0-388.el9.x86_6 18 MB/s | 32 MB 00:01 2023-12-02T02:33:36.525 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:33:36.525 INFO:teuthology.orchestra.run.smithi029.stdout:Total 43 MB/s | 96 MB 00:02 2023-12-02T02:33:37.412 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-02T02:33:37.866 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-02T02:33:37.866 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-02T02:33:39.219 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-02T02:33:39.219 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-02T02:33:39.404 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2023-12-02T02:33:40.109 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : kernel-modules-core-5.14.0-388.el9.x86_64 1/4 2023-12-02T02:33:40.258 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-02T02:33:42.038 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-02T02:33:42.798 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-core-5.14.0-388.el9.x86_64 1/4 2023-12-02T02:33:42.980 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-02T02:33:43.170 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-02T02:33:43.352 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-02T02:33:45.943 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-02T02:33:46.215 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-02T02:33:48.313 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-02T02:33:48.436 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : kernel-5.14.0-388.el9.x86_64 4/4 2023-12-02T02:33:51.177 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-02T02:33:51.339 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-5.14.0-388.el9.x86_64 4/4 2023-12-02T02:33:53.415 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-02T02:33:56.325 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-02T02:34:23.998 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 4/4 2023-12-02T02:34:24.043 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 4/4 2023-12-02T02:34:24.417 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-5.14.0-388.el9.x86_64 4/4 2023-12-02T02:34:24.418 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : kernel-5.14.0-388.el9.x86_64 1/4 2023-12-02T02:34:24.418 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-02T02:34:24.418 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-02T02:34:24.843 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-02T02:34:24.844 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:34:24.844 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2023-12-02T02:34:24.844 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-5.14.0-388.el9.x86_64 2023-12-02T02:34:24.844 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-core-5.14.0-388.el9.x86_64 2023-12-02T02:34:24.844 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-modules-5.14.0-388.el9.x86_64 2023-12-02T02:34:24.844 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-modules-core-5.14.0-388.el9.x86_64 2023-12-02T02:34:24.844 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:34:24.844 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-12-02T02:34:25.329 DEBUG:teuthology.orchestra.run.smithi057:> rpm -q kernel | sort -rV | head -n 1 2023-12-02T02:34:25.469 INFO:teuthology.orchestra.run.smithi057.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-02T02:34:25.469 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-02T02:34:25.469 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-388.el9.x86_64 2023-12-02T02:34:25.469 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-02T02:34:25.469 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-02T02:34:25.470 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2023-12-02T02:34:25.470 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi057.front.sepia.ceph.com, path=None, version=distro) 2023-12-02T02:34:25.470 DEBUG:teuthology.orchestra.run.smithi057:> rpm -q kernel | sort -rV | head -n 1 2023-12-02T02:34:25.593 INFO:teuthology.orchestra.run.smithi057.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-02T02:34:25.593 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-02T02:34:25.593 DEBUG:teuthology.orchestra.run.smithi057:> sudo rpm -qi grub2-tools 2023-12-02T02:34:25.641 INFO:teuthology.orchestra.run.smithi057.stdout:Name : grub2-tools 2023-12-02T02:34:25.641 INFO:teuthology.orchestra.run.smithi057.stdout:Epoch : 1 2023-12-02T02:34:25.641 INFO:teuthology.orchestra.run.smithi057.stdout:Version : 2.06 2023-12-02T02:34:25.642 INFO:teuthology.orchestra.run.smithi057.stdout:Release : 68.el9 2023-12-02T02:34:25.642 INFO:teuthology.orchestra.run.smithi057.stdout:Architecture: x86_64 2023-12-02T02:34:25.642 INFO:teuthology.orchestra.run.smithi057.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-02T02:34:25.642 INFO:teuthology.orchestra.run.smithi057.stdout:Group : Unspecified 2023-12-02T02:34:25.642 INFO:teuthology.orchestra.run.smithi057.stdout:Size : 8264254 2023-12-02T02:34:25.642 INFO:teuthology.orchestra.run.smithi057.stdout:License : GPLv3+ 2023-12-02T02:34:25.642 INFO:teuthology.orchestra.run.smithi057.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-02T02:34:25.642 INFO:teuthology.orchestra.run.smithi057.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-02T02:34:25.642 INFO:teuthology.orchestra.run.smithi057.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-02T02:34:25.642 INFO:teuthology.orchestra.run.smithi057.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-02T02:34:25.642 INFO:teuthology.orchestra.run.smithi057.stdout:Packager : builder@centos.org 2023-12-02T02:34:25.642 INFO:teuthology.orchestra.run.smithi057.stdout:Vendor : CentOS 2023-12-02T02:34:25.642 INFO:teuthology.orchestra.run.smithi057.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-02T02:34:25.642 INFO:teuthology.orchestra.run.smithi057.stdout:Summary : Support tools for GRUB. 2023-12-02T02:34:25.643 INFO:teuthology.orchestra.run.smithi057.stdout:Description : 2023-12-02T02:34:25.643 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:34:25.643 INFO:teuthology.orchestra.run.smithi057.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-02T02:34:25.643 INFO:teuthology.orchestra.run.smithi057.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-02T02:34:25.643 INFO:teuthology.orchestra.run.smithi057.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-02T02:34:25.643 INFO:teuthology.orchestra.run.smithi057.stdout:hardware devices. 2023-12-02T02:34:25.643 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:34:25.643 INFO:teuthology.orchestra.run.smithi057.stdout:This subpackage provides tools for support of all platforms. 2023-12-02T02:34:25.644 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-02T02:34:25.644 INFO:teuthology.task.kernel:Updating grub on smithi057 to boot 5.14.0-388.el9.x86_64 2023-12-02T02:34:25.644 DEBUG:teuthology.orchestra.run.smithi057:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-02T02:34:26.270 INFO:teuthology.orchestra.run.smithi057.stderr:Generating grub configuration file ... 2023-12-02T02:34:27.624 INFO:teuthology.orchestra.run.smithi057.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-02T02:34:27.645 INFO:teuthology.orchestra.run.smithi057.stderr:done 2023-12-02T02:34:27.647 DEBUG:teuthology.orchestra.run.smithi057:> mktemp 2023-12-02T02:34:27.663 INFO:teuthology.orchestra.run.smithi057.stdout:/tmp/tmp.XAzDY25LFj 2023-12-02T02:34:27.663 DEBUG:teuthology.orchestra.run.smithi057:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.XAzDY25LFj 2023-12-02T02:34:27.736 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0666 /tmp/tmp.XAzDY25LFj 2023-12-02T02:34:27.877 DEBUG:teuthology.orchestra.remote:smithi057:/tmp/tmp.XAzDY25LFj is 6KB 2023-12-02T02:34:27.925 DEBUG:teuthology.orchestra.run.smithi057:> rm -fr /tmp/tmp.XAzDY25LFj 2023-12-02T02:34:27.940 DEBUG:teuthology.orchestra.run.smithi057:> sudo /bin/ls /boot/loader/entries || true 2023-12-02T02:34:28.008 INFO:teuthology.orchestra.run.smithi057.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-02T02:34:28.008 INFO:teuthology.orchestra.run.smithi057.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-02T02:34:28.008 INFO:teuthology.orchestra.run.smithi057.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-02T02:34:28.008 INFO:teuthology.orchestra.run.smithi057.stdout:15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64.conf 2023-12-02T02:34:28.009 DEBUG:teuthology.orchestra.run.smithi057:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64 2023-12-02T02:34:28.156 DEBUG:teuthology.orchestra.run.smithi057:> sudo shutdown -r now 2023-12-02T02:34:29.070 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 4/4 2023-12-02T02:34:29.116 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 4/4 2023-12-02T02:34:29.524 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-5.14.0-388.el9.x86_64 4/4 2023-12-02T02:34:29.524 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-5.14.0-388.el9.x86_64 1/4 2023-12-02T02:34:29.524 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-02T02:34:29.524 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-02T02:34:29.959 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-02T02:34:29.959 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:34:29.959 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-02T02:34:29.959 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-5.14.0-388.el9.x86_64 2023-12-02T02:34:29.959 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core-5.14.0-388.el9.x86_64 2023-12-02T02:34:29.960 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-5.14.0-388.el9.x86_64 2023-12-02T02:34:29.960 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core-5.14.0-388.el9.x86_64 2023-12-02T02:34:29.960 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:34:29.960 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-02T02:34:30.675 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2023-12-02T02:34:30.840 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-02T02:34:30.841 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-02T02:34:30.841 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-388.el9.x86_64 2023-12-02T02:34:30.841 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-02T02:34:30.841 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-02T02:34:30.841 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2023-12-02T02:34:30.841 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi029.front.sepia.ceph.com, path=None, version=distro) 2023-12-02T02:34:30.841 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2023-12-02T02:34:30.964 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-02T02:34:30.964 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-02T02:34:30.965 DEBUG:teuthology.orchestra.run.smithi029:> sudo rpm -qi grub2-tools 2023-12-02T02:34:31.001 INFO:teuthology.orchestra.run.smithi029.stdout:Name : grub2-tools 2023-12-02T02:34:31.002 INFO:teuthology.orchestra.run.smithi029.stdout:Epoch : 1 2023-12-02T02:34:31.002 INFO:teuthology.orchestra.run.smithi029.stdout:Version : 2.06 2023-12-02T02:34:31.002 INFO:teuthology.orchestra.run.smithi029.stdout:Release : 68.el9 2023-12-02T02:34:31.002 INFO:teuthology.orchestra.run.smithi029.stdout:Architecture: x86_64 2023-12-02T02:34:31.002 INFO:teuthology.orchestra.run.smithi029.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-02T02:34:31.002 INFO:teuthology.orchestra.run.smithi029.stdout:Group : Unspecified 2023-12-02T02:34:31.002 INFO:teuthology.orchestra.run.smithi029.stdout:Size : 8264254 2023-12-02T02:34:31.002 INFO:teuthology.orchestra.run.smithi029.stdout:License : GPLv3+ 2023-12-02T02:34:31.002 INFO:teuthology.orchestra.run.smithi029.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-02T02:34:31.002 INFO:teuthology.orchestra.run.smithi029.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-02T02:34:31.002 INFO:teuthology.orchestra.run.smithi029.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-02T02:34:31.002 INFO:teuthology.orchestra.run.smithi029.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-02T02:34:31.002 INFO:teuthology.orchestra.run.smithi029.stdout:Packager : builder@centos.org 2023-12-02T02:34:31.002 INFO:teuthology.orchestra.run.smithi029.stdout:Vendor : CentOS 2023-12-02T02:34:31.003 INFO:teuthology.orchestra.run.smithi029.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-02T02:34:31.003 INFO:teuthology.orchestra.run.smithi029.stdout:Summary : Support tools for GRUB. 2023-12-02T02:34:31.003 INFO:teuthology.orchestra.run.smithi029.stdout:Description : 2023-12-02T02:34:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:34:31.003 INFO:teuthology.orchestra.run.smithi029.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-02T02:34:31.003 INFO:teuthology.orchestra.run.smithi029.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-02T02:34:31.003 INFO:teuthology.orchestra.run.smithi029.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-02T02:34:31.003 INFO:teuthology.orchestra.run.smithi029.stdout:hardware devices. 2023-12-02T02:34:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:34:31.003 INFO:teuthology.orchestra.run.smithi029.stdout:This subpackage provides tools for support of all platforms. 2023-12-02T02:34:31.004 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-02T02:34:31.004 INFO:teuthology.task.kernel:Updating grub on smithi029 to boot 5.14.0-388.el9.x86_64 2023-12-02T02:34:31.004 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-02T02:34:31.724 INFO:teuthology.orchestra.run.smithi029.stderr:Generating grub configuration file ... 2023-12-02T02:34:33.310 INFO:teuthology.orchestra.run.smithi029.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-02T02:34:33.335 INFO:teuthology.orchestra.run.smithi029.stderr:done 2023-12-02T02:34:33.337 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2023-12-02T02:34:33.352 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.sEe6drNQqX 2023-12-02T02:34:33.352 DEBUG:teuthology.orchestra.run.smithi029:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.sEe6drNQqX 2023-12-02T02:34:33.429 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.sEe6drNQqX 2023-12-02T02:34:33.574 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.sEe6drNQqX is 6KB 2023-12-02T02:34:33.625 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.sEe6drNQqX 2023-12-02T02:34:33.640 DEBUG:teuthology.orchestra.run.smithi029:> sudo /bin/ls /boot/loader/entries || true 2023-12-02T02:34:33.708 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-02T02:34:33.708 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-02T02:34:33.708 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-02T02:34:33.708 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64.conf 2023-12-02T02:34:33.709 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64 2023-12-02T02:34:33.861 DEBUG:teuthology.orchestra.run.smithi029:> sudo shutdown -r now 2023-12-02T02:34:58.182 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2023-12-02T02:34:58.183 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-12-02T02:34:58.183 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-02T02:35:03.866 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2023-12-02T02:35:03.866 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-12-02T02:35:03.867 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-02T02:35:22.338 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.29 2023-12-02T02:35:31.346 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-12-02T02:35:31.347 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-02T02:35:32.834 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.57 2023-12-02T02:35:41.842 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-12-02T02:35:41.843 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-02T02:36:31.398 DEBUG:teuthology.orchestra.remote:timed out 2023-12-02T02:36:41.854 DEBUG:teuthology.orchestra.remote:timed out 2023-12-02T02:36:43.401 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-12-02T02:36:43.402 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-02T02:36:43.816 DEBUG:teuthology.orchestra.run.smithi029:> true 2023-12-02T02:36:44.219 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-12-02T02:36:44.219 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.14.0-388.el9.x86_64"... 2023-12-02T02:36:44.219 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2023-12-02T02:36:44.235 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-388.el9.x86_64 2023-12-02T02:36:44.236 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-388.el9.x86_64 vs 5.14.0-388.el9.x86_64 2023-12-02T02:36:44.236 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-02T02:36:44.236 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-02T02:36:45.237 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2023-12-02T02:36:45.237 DEBUG:teuthology.orchestra.run.smithi029:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-02T02:36:45.328 INFO:teuthology.orchestra.run.smithi029.stdout:ttyS1 2023-12-02T02:36:45.367 DEBUG:teuthology.parallel:result is None 2023-12-02T02:36:53.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-12-02T02:36:53.856 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-02T02:36:54.268 DEBUG:teuthology.orchestra.run.smithi057:> true 2023-12-02T02:36:54.654 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-12-02T02:36:54.654 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.14.0-388.el9.x86_64"... 2023-12-02T02:36:54.654 DEBUG:teuthology.orchestra.run.smithi057:> uname -r 2023-12-02T02:36:54.669 INFO:teuthology.orchestra.run.smithi057.stdout:5.14.0-388.el9.x86_64 2023-12-02T02:36:54.670 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-388.el9.x86_64 vs 5.14.0-388.el9.x86_64 2023-12-02T02:36:54.670 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-02T02:36:54.670 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-02T02:36:55.670 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2023-12-02T02:36:55.671 DEBUG:teuthology.orchestra.run.smithi057:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-02T02:36:55.745 INFO:teuthology.orchestra.run.smithi057.stdout:ttyS1 2023-12-02T02:36:55.783 DEBUG:teuthology.parallel:result is None 2023-12-02T02:36:55.784 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-02T02:36:55.791 INFO:teuthology.task.internal:Creating test directory... 2023-12-02T02:36:55.791 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-02T02:36:55.794 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-02T02:36:55.811 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-02T02:36:55.842 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-02T02:36:55.926 INFO:teuthology.task.internal:Creating archive directory... 2023-12-02T02:36:55.926 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-02T02:36:55.929 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-02T02:36:55.963 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-02T02:36:56.001 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-02T02:36:56.001 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-02T02:36:56.004 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-02T02:36:56.046 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T02:36:56.048 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T02:36:56.064 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T02:36:56.066 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-02T02:36:56.068 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-02T02:36:56.085 INFO:teuthology.task.internal:Configuring sudo... 2023-12-02T02:36:56.085 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-02T02:36:56.087 DEBUG:teuthology.orchestra.run.smithi057:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-02T02:36:56.140 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-02T02:36:56.153 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-02T02:36:56.153 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-02T02:36:56.162 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-02T02:36:56.196 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-02T02:36:56.260 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-02T02:36:56.326 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-02T02:36:56.369 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-02T02:36:56.416 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-02T02:36:56.416 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-02T02:36:56.488 DEBUG:teuthology.orchestra.run.smithi057:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-02T02:36:56.532 DEBUG:teuthology.orchestra.run.smithi057:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-02T02:36:56.587 DEBUG:teuthology.orchestra.run.smithi057:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-02T02:36:56.671 DEBUG:teuthology.orchestra.run.smithi057:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-02T02:36:56.719 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:36:56.719 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-02T02:36:56.792 DEBUG:teuthology.orchestra.run.smithi029:> sudo service rsyslog restart 2023-12-02T02:36:56.794 DEBUG:teuthology.orchestra.run.smithi057:> sudo service rsyslog restart 2023-12-02T02:36:56.837 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-02T02:36:56.880 INFO:teuthology.orchestra.run.smithi057.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-02T02:36:57.307 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-02T02:36:57.315 INFO:teuthology.task.internal:Starting timer... 2023-12-02T02:36:57.316 INFO:teuthology.run_tasks:Running task pcp... 2023-12-02T02:36:57.346 INFO:teuthology.run_tasks:Running task selinux... 2023-12-02T02:36:57.394 DEBUG:teuthology.orchestra.run.smithi029:> sudo service auditd rotate 2023-12-02T02:36:57.472 INFO:teuthology.orchestra.run.smithi029.stdout:Rotating logs: 2023-12-02T02:36:57.474 DEBUG:teuthology.orchestra.run.smithi057:> sudo service auditd rotate 2023-12-02T02:36:57.537 INFO:teuthology.orchestra.run.smithi057.stdout:Rotating logs: 2023-12-02T02:36:57.539 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-02T02:36:57.539 DEBUG:teuthology.orchestra.run.smithi029:> /usr/sbin/getenforce 2023-12-02T02:36:57.567 INFO:teuthology.orchestra.run.smithi029.stdout:Permissive 2023-12-02T02:36:57.567 DEBUG:teuthology.orchestra.run.smithi057:> /usr/sbin/getenforce 2023-12-02T02:36:57.602 INFO:teuthology.orchestra.run.smithi057.stdout:Permissive 2023-12-02T02:36:57.602 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi029.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi057.front.sepia.ceph.com': 'permissive'} 2023-12-02T02:36:57.602 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-02T02:36:57.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-02T02:36:57.640 DEBUG:teuthology.orchestra.run.smithi057:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-02T02:36:57.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-02T02:36:57.673 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-02T02:36:57.674 DEBUG:teuthology.orchestra.run.smithi029:> sudo /usr/sbin/setenforce permissive 2023-12-02T02:36:57.719 DEBUG:teuthology.orchestra.run.smithi057:> sudo /usr/sbin/setenforce permissive 2023-12-02T02:36:57.754 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-02T02:36:57.765 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-02T02:36:57.765 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-02T02:36:57.784 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-02T02:36:57.785 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi029.front.sepia.ceph.com,smithi057.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-02T02:44:17.874 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi029.front.sepia.ceph.com'), Remote(name='ubuntu@smithi057.front.sepia.ceph.com')] 2023-12-02T02:44:17.875 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-12-02T02:44:17.875 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-02T02:44:17.973 DEBUG:teuthology.orchestra.run.smithi029:> true 2023-12-02T02:44:18.041 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-12-02T02:44:18.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-12-02T02:44:18.042 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-02T02:44:18.140 DEBUG:teuthology.orchestra.run.smithi057:> true 2023-12-02T02:44:18.211 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-12-02T02:44:18.211 INFO:teuthology.run_tasks:Running task clock... 2023-12-02T02:44:18.222 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-02T02:44:18.222 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-02T02:44:18.223 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-02T02:44:18.225 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-02T02:44:18.225 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-02T02:44:18.258 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-02T02:44:18.278 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-02T02:44:18.289 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-02T02:44:18.310 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-02T02:44:18.315 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: ntpd: command not found 2023-12-02T02:44:18.332 INFO:teuthology.orchestra.run.smithi029.stdout:506 Cannot talk to daemon 2023-12-02T02:44:18.347 INFO:teuthology.orchestra.run.smithi057.stderr:sudo: ntpd: command not found 2023-12-02T02:44:18.352 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-02T02:44:18.364 INFO:teuthology.orchestra.run.smithi057.stdout:506 Cannot talk to daemon 2023-12-02T02:44:18.373 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-02T02:44:18.384 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-02T02:44:18.404 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-02T02:44:18.421 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2023-12-02T02:44:18.424 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-02T02:44:18.424 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2023-12-02T02:44:18.424 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-02T02:44:18.424 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-02T02:44:18.424 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-02T02:44:18.424 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-02T02:44:18.450 INFO:teuthology.orchestra.run.smithi057.stderr:bash: line 1: ntpq: command not found 2023-12-02T02:44:18.454 INFO:teuthology.orchestra.run.smithi057.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-02T02:44:18.454 INFO:teuthology.orchestra.run.smithi057.stdout:=============================================================================== 2023-12-02T02:44:18.454 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-02T02:44:18.454 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-02T02:44:18.454 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-02T02:44:18.454 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-02T02:44:18.455 INFO:teuthology.run_tasks:Running task install... 2023-12-02T02:44:18.463 DEBUG:teuthology.task.install:project ceph 2023-12-02T02:44:18.463 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bed7681480ccb0192b91a35e691658867e1e96b7'}} 2023-12-02T02:44:18.463 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'bed7681480ccb0192b91a35e691658867e1e96b7'} 2023-12-02T02:44:18.463 INFO:teuthology.task.install:Using flavor: default 2023-12-02T02:44:18.469 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-02T02:44:18.469 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2023-12-02T02:44:18.469 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bed7681480ccb0192b91a35e691658867e1e96b7', 'tag': None, 'wait_for_package': False} 2023-12-02T02:44:18.470 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bed7681480ccb0192b91a35e691658867e1e96b7 2023-12-02T02:44:18.471 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bed7681480ccb0192b91a35e691658867e1e96b7', 'tag': None, 'wait_for_package': False} 2023-12-02T02:44:18.471 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bed7681480ccb0192b91a35e691658867e1e96b7 2023-12-02T02:44:18.615 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main/bed7681480ccb0192b91a35e691658867e1e96b7/centos/9/flavors/default/ 2023-12-02T02:44:18.615 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7631.gbed76814 2023-12-02T02:44:18.623 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main/bed7681480ccb0192b91a35e691658867e1e96b7/centos/9/flavors/default/ 2023-12-02T02:44:18.623 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7631.gbed76814 2023-12-02T02:44:18.730 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main/bed7681480ccb0192b91a35e691658867e1e96b7/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main/bed7681480ccb0192b91a35e691658867e1e96b7/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main/bed7681480ccb0192b91a35e691658867e1e96b7/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-02T02:44:18.731 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:44:18.731 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-02T02:44:18.763 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2023-12-02T02:44:18.763 DEBUG:teuthology.orchestra.run.smithi057:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bed7681480ccb0192b91a35e691658867e1e96b7/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-02T02:44:18.794 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main/bed7681480ccb0192b91a35e691658867e1e96b7/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main/bed7681480ccb0192b91a35e691658867e1e96b7/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main/bed7681480ccb0192b91a35e691658867e1e96b7/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-02T02:44:18.795 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-02T02:44:18.795 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-02T02:44:18.828 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2023-12-02T02:44:18.828 DEBUG:teuthology.orchestra.run.smithi029:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bed7681480ccb0192b91a35e691658867e1e96b7/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-02T02:44:18.835 DEBUG:teuthology.orchestra.run.smithi057:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-02T02:44:18.900 DEBUG:teuthology.orchestra.run.smithi029:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-02T02:44:18.923 DEBUG:teuthology.orchestra.run.smithi057:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-02T02:44:18.953 INFO:teuthology.orchestra.run.smithi057.stdout:check_obsoletes = 1 2023-12-02T02:44:18.954 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum clean all 2023-12-02T02:44:18.988 DEBUG:teuthology.orchestra.run.smithi029:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-02T02:44:19.018 INFO:teuthology.orchestra.run.smithi029.stdout:check_obsoletes = 1 2023-12-02T02:44:19.020 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum clean all 2023-12-02T02:44:19.217 INFO:teuthology.orchestra.run.smithi057.stdout:73 files removed 2023-12-02T02:44:19.244 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-radosgw 2023-12-02T02:44:19.267 INFO:teuthology.orchestra.run.smithi029.stdout:73 files removed 2023-12-02T02:44:19.295 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-radosgw 2023-12-02T02:44:19.994 INFO:teuthology.orchestra.run.smithi057.stdout:ceph packages for x86_64 189 kB/s | 78 kB 00:00 2023-12-02T02:44:20.141 INFO:teuthology.orchestra.run.smithi029.stdout:ceph packages for x86_64 154 kB/s | 78 kB 00:00 2023-12-02T02:44:20.336 INFO:teuthology.orchestra.run.smithi057.stdout:ceph noarch packages 32 kB/s | 9.5 kB 00:00 2023-12-02T02:44:20.652 INFO:teuthology.orchestra.run.smithi029.stdout:ceph noarch packages 21 kB/s | 9.5 kB 00:00 2023-12-02T02:44:20.653 INFO:teuthology.orchestra.run.smithi057.stdout:ceph source packages 6.7 kB/s | 1.9 kB 00:00 2023-12-02T02:44:21.295 INFO:teuthology.orchestra.run.smithi029.stdout:ceph source packages 3.1 kB/s | 1.9 kB 00:00 2023-12-02T02:44:21.489 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS-9 - AppStream 8.0 MB/s | 6.4 MB 00:00 2023-12-02T02:44:21.765 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - AppStream 15 MB/s | 6.4 MB 00:00 2023-12-02T02:44:23.233 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - BaseOS 5.7 MB/s | 2.0 MB 00:00 2023-12-02T02:44:23.423 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS-9 - BaseOS 2.4 MB/s | 2.0 MB 00:00 2023-12-02T02:44:26.080 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - BaseOS 3.4 MB/s | 7.9 MB 00:02 2023-12-02T02:44:28.232 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 1.7 MB/s | 7.9 MB 00:04 2023-12-02T02:44:28.823 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - AppStream 11 MB/s | 18 MB 00:01 2023-12-02T02:44:33.741 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 5.8 MB 00:01 2023-12-02T02:44:33.869 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 4.1 MB/s | 18 MB 00:04 2023-12-02T02:44:35.147 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 15 kB 00:00 2023-12-02T02:44:36.183 INFO:teuthology.orchestra.run.smithi057.stdout:Extra Packages for Enterprise Linux 22 MB/s | 20 MB 00:00 2023-12-02T02:44:37.913 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 11 MB/s | 5.8 MB 00:00 2023-12-02T02:44:39.362 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 15 kB 00:00 2023-12-02T02:44:41.058 INFO:teuthology.orchestra.run.smithi057.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2023-12-02T02:44:43.877 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-12-02T02:44:43.881 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:44:43.881 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2023-12-02T02:44:43.882 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:44:43.882 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2023-12-02T02:44:43.882 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-radosgw x86_64 2:18.0.0-7631.gbed76814.el9 ceph 8.6 M 2023-12-02T02:44:43.882 INFO:teuthology.orchestra.run.smithi057.stdout:Upgrading: 2023-12-02T02:44:43.882 INFO:teuthology.orchestra.run.smithi057.stdout: librados2 x86_64 2:18.0.0-7631.gbed76814.el9 ceph 3.3 M 2023-12-02T02:44:43.882 INFO:teuthology.orchestra.run.smithi057.stdout: librbd1 x86_64 2:18.0.0-7631.gbed76814.el9 ceph 3.0 M 2023-12-02T02:44:43.882 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2023-12-02T02:44:43.883 INFO:teuthology.orchestra.run.smithi057.stdout: boost-program-options 2023-12-02T02:44:43.883 INFO:teuthology.orchestra.run.smithi057.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-02T02:44:43.883 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-base x86_64 2:18.0.0-7631.gbed76814.el9 ceph 5.2 M 2023-12-02T02:44:43.883 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-common x86_64 2:18.0.0-7631.gbed76814.el9 ceph 19 M 2023-12-02T02:44:43.883 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-selinux x86_64 2:18.0.0-7631.gbed76814.el9 ceph 24 k 2023-12-02T02:44:43.883 INFO:teuthology.orchestra.run.smithi057.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-02T02:44:43.883 INFO:teuthology.orchestra.run.smithi057.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-12-02T02:44:43.883 INFO:teuthology.orchestra.run.smithi057.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-12-02T02:44:43.883 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs2 x86_64 2:18.0.0-7631.gbed76814.el9 ceph 718 k 2023-12-02T02:44:43.883 INFO:teuthology.orchestra.run.smithi057.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-02T02:44:43.883 INFO:teuthology.orchestra.run.smithi057.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-02T02:44:43.884 INFO:teuthology.orchestra.run.smithi057.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-02T02:44:43.884 INFO:teuthology.orchestra.run.smithi057.stdout: libradosstriper1 x86_64 2:18.0.0-7631.gbed76814.el9 ceph 475 k 2023-12-02T02:44:43.884 INFO:teuthology.orchestra.run.smithi057.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-02T02:44:43.884 INFO:teuthology.orchestra.run.smithi057.stdout: librgw2 x86_64 2:18.0.0-7631.gbed76814.el9 ceph 4.8 M 2023-12-02T02:44:43.884 INFO:teuthology.orchestra.run.smithi057.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-02T02:44:43.884 INFO:teuthology.orchestra.run.smithi057.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-02T02:44:43.884 INFO:teuthology.orchestra.run.smithi057.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-12-02T02:44:43.884 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-argparse 2023-12-02T02:44:43.884 INFO:teuthology.orchestra.run.smithi057.stdout: x86_64 2:18.0.0-7631.gbed76814.el9 ceph 45 k 2023-12-02T02:44:43.884 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-common x86_64 2:18.0.0-7631.gbed76814.el9 ceph 129 k 2023-12-02T02:44:43.884 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cephfs x86_64 2:18.0.0-7631.gbed76814.el9 ceph 162 k 2023-12-02T02:44:43.884 INFO:teuthology.orchestra.run.smithi057.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-02T02:44:43.884 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rados x86_64 2:18.0.0-7631.gbed76814.el9 ceph 320 k 2023-12-02T02:44:43.884 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rbd x86_64 2:18.0.0-7631.gbed76814.el9 ceph 298 k 2023-12-02T02:44:43.885 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rgw x86_64 2:18.0.0-7631.gbed76814.el9 ceph 99 k 2023-12-02T02:44:43.885 INFO:teuthology.orchestra.run.smithi057.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-02T02:44:43.885 INFO:teuthology.orchestra.run.smithi057.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-02T02:44:43.885 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:44:43.885 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2023-12-02T02:44:43.885 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:44:43.885 INFO:teuthology.orchestra.run.smithi057.stdout:Install 27 Packages 2023-12-02T02:44:43.885 INFO:teuthology.orchestra.run.smithi057.stdout:Upgrade 2 Packages 2023-12-02T02:44:43.885 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:44:43.886 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 55 M 2023-12-02T02:44:43.886 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2023-12-02T02:44:43.995 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 4.4 MB/s | 20 MB 00:04 2023-12-02T02:44:48.960 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2023-12-02T02:44:49.292 INFO:teuthology.orchestra.run.smithi057.stdout:(1/29): ceph-base-18.0.0-7631.gbed76814.el9.x86 1.0 MB/s | 5.2 MB 00:05 2023-12-02T02:44:49.409 INFO:teuthology.orchestra.run.smithi057.stdout:(2/29): ceph-selinux-18.0.0-7631.gbed76814.el9. 207 kB/s | 24 kB 00:00 2023-12-02T02:44:49.526 INFO:teuthology.orchestra.run.smithi057.stdout:(3/29): ceph-radosgw-18.0.0-7631.gbed76814.el9. 1.6 MB/s | 8.6 MB 00:05 2023-12-02T02:44:49.851 INFO:teuthology.orchestra.run.smithi057.stdout:(4/29): libradosstriper1-18.0.0-7631.gbed76814. 1.4 MB/s | 475 kB 00:00 2023-12-02T02:44:50.177 INFO:teuthology.orchestra.run.smithi057.stdout:(5/29): libcephfs2-18.0.0-7631.gbed76814.el9.x8 936 kB/s | 718 kB 00:00 2023-12-02T02:44:50.293 INFO:teuthology.orchestra.run.smithi057.stdout:(6/29): python3-ceph-argparse-18.0.0-7631.gbed7 387 kB/s | 45 kB 00:00 2023-12-02T02:44:50.494 INFO:teuthology.orchestra.run.smithi057.stdout:(7/29): python3-ceph-common-18.0.0-7631.gbed768 645 kB/s | 129 kB 00:00 2023-12-02T02:44:50.702 INFO:teuthology.orchestra.run.smithi057.stdout:(8/29): python3-cephfs-18.0.0-7631.gbed76814.el 778 kB/s | 162 kB 00:00 2023-12-02T02:44:50.986 INFO:teuthology.orchestra.run.smithi057.stdout:(9/29): python3-rados-18.0.0-7631.gbed76814.el9 1.1 MB/s | 320 kB 00:00 2023-12-02T02:44:51.303 INFO:teuthology.orchestra.run.smithi057.stdout:(10/29): python3-rbd-18.0.0-7631.gbed76814.el9. 943 kB/s | 298 kB 00:00 2023-12-02T02:44:51.436 INFO:teuthology.orchestra.run.smithi057.stdout:(11/29): python3-rgw-18.0.0-7631.gbed76814.el9. 745 kB/s | 99 kB 00:00 2023-12-02T02:44:51.570 INFO:teuthology.orchestra.run.smithi057.stdout:(12/29): boost-program-options-1.75.0-8.el9.x86 794 kB/s | 106 kB 00:00 2023-12-02T02:44:51.612 INFO:teuthology.orchestra.run.smithi057.stdout:(13/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 159 kB 00:00 2023-12-02T02:44:51.637 INFO:teuthology.orchestra.run.smithi057.stdout:(14/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2023-12-02T02:44:51.696 INFO:teuthology.orchestra.run.smithi057.stdout:(15/29): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2023-12-02T02:44:51.730 INFO:teuthology.orchestra.run.smithi057.stdout:(16/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 8.3 MB/s | 282 kB 00:00 2023-12-02T02:44:51.755 INFO:teuthology.orchestra.run.smithi057.stdout:(17/29): python3-prettytable-0.7.2-27.el9.noarc 1.6 MB/s | 41 kB 00:00 2023-12-02T02:44:51.839 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-02T02:44:51.861 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:44:51.861 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-12-02T02:44:51.861 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:44:51.862 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-02T02:44:51.862 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw x86_64 2:18.0.0-7631.gbed76814.el9 ceph 8.6 M 2023-12-02T02:44:51.862 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrading: 2023-12-02T02:44:51.862 INFO:teuthology.orchestra.run.smithi029.stdout: librados2 x86_64 2:18.0.0-7631.gbed76814.el9 ceph 3.3 M 2023-12-02T02:44:51.862 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1 x86_64 2:18.0.0-7631.gbed76814.el9 ceph 3.0 M 2023-12-02T02:44:51.862 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-12-02T02:44:51.862 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options 2023-12-02T02:44:51.862 INFO:teuthology.orchestra.run.smithi029.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-02T02:44:51.862 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base x86_64 2:18.0.0-7631.gbed76814.el9 ceph 5.2 M 2023-12-02T02:44:51.862 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common x86_64 2:18.0.0-7631.gbed76814.el9 ceph 19 M 2023-12-02T02:44:51.862 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux x86_64 2:18.0.0-7631.gbed76814.el9 ceph 24 k 2023-12-02T02:44:51.862 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-02T02:44:51.862 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-12-02T02:44:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-12-02T02:44:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2 x86_64 2:18.0.0-7631.gbed76814.el9 ceph 718 k 2023-12-02T02:44:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-02T02:44:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-02T02:44:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-02T02:44:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1 x86_64 2:18.0.0-7631.gbed76814.el9 ceph 475 k 2023-12-02T02:44:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-02T02:44:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2 x86_64 2:18.0.0-7631.gbed76814.el9 ceph 4.8 M 2023-12-02T02:44:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-02T02:44:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-02T02:44:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-12-02T02:44:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse 2023-12-02T02:44:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: x86_64 2:18.0.0-7631.gbed76814.el9 ceph 45 k 2023-12-02T02:44:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common x86_64 2:18.0.0-7631.gbed76814.el9 ceph 129 k 2023-12-02T02:44:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs x86_64 2:18.0.0-7631.gbed76814.el9 ceph 162 k 2023-12-02T02:44:51.864 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-02T02:44:51.864 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados x86_64 2:18.0.0-7631.gbed76814.el9 ceph 320 k 2023-12-02T02:44:51.864 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd x86_64 2:18.0.0-7631.gbed76814.el9 ceph 298 k 2023-12-02T02:44:51.864 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw x86_64 2:18.0.0-7631.gbed76814.el9 ceph 99 k 2023-12-02T02:44:51.864 INFO:teuthology.orchestra.run.smithi029.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-02T02:44:51.864 INFO:teuthology.orchestra.run.smithi029.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-02T02:44:51.864 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:44:51.864 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-02T02:44:51.864 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:44:51.864 INFO:teuthology.orchestra.run.smithi029.stdout:Install 27 Packages 2023-12-02T02:44:51.864 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrade 2 Packages 2023-12-02T02:44:51.864 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:44:51.865 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 55 M 2023-12-02T02:44:51.865 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-02T02:44:52.113 INFO:teuthology.orchestra.run.smithi057.stdout:(18/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 862 kB/s | 309 kB 00:00 2023-12-02T02:44:52.414 INFO:teuthology.orchestra.run.smithi057.stdout:(19/29): libarrow-9.0.0-9.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2023-12-02T02:44:52.481 INFO:teuthology.orchestra.run.smithi057.stdout:(20/29): librgw2-18.0.0-7631.gbed76814.el9.x86_ 1.8 MB/s | 4.8 MB 00:02 2023-12-02T02:44:52.498 INFO:teuthology.orchestra.run.smithi057.stdout:(21/29): libarrow-doc-9.0.0-9.el9.noarch.rpm 291 kB/s | 24 kB 00:00 2023-12-02T02:44:52.581 INFO:teuthology.orchestra.run.smithi057.stdout:(22/29): libunwind-1.6.2-1.el9.x86_64.rpm 809 kB/s | 67 kB 00:00 2023-12-02T02:44:52.707 INFO:teuthology.orchestra.run.smithi057.stdout:(23/29): liboath-2.6.7-2.el9.x86_64.rpm 215 kB/s | 49 kB 00:00 2023-12-02T02:44:52.740 INFO:teuthology.orchestra.run.smithi057.stdout:(24/29): parquet-libs-9.0.0-9.el9.x86_64.rpm 5.2 MB/s | 837 kB 00:00 2023-12-02T02:44:52.933 INFO:teuthology.orchestra.run.smithi057.stdout:(25/29): thrift-0.15.0-2.el9.x86_64.rpm 8.2 MB/s | 1.6 MB 00:00 2023-12-02T02:44:52.958 INFO:teuthology.orchestra.run.smithi057.stdout:(26/29): re2-20211101-3.el9.x86_64.rpm 764 kB/s | 192 kB 00:00 2023-12-02T02:44:54.508 INFO:teuthology.orchestra.run.smithi057.stdout:(27/29): librbd1-18.0.0-7631.gbed76814.el9.x86_ 1.9 MB/s | 3.0 MB 00:01 2023-12-02T02:44:54.879 INFO:teuthology.orchestra.run.smithi029.stdout:(1/29): ceph-base-18.0.0-7631.gbed76814.el9.x86 1.8 MB/s | 5.2 MB 00:02 2023-12-02T02:44:54.971 INFO:teuthology.orchestra.run.smithi029.stdout:(2/29): ceph-selinux-18.0.0-7631.gbed76814.el9. 262 kB/s | 24 kB 00:00 2023-12-02T02:44:55.263 INFO:teuthology.orchestra.run.smithi029.stdout:(3/29): libcephfs2-18.0.0-7631.gbed76814.el9.x8 2.4 MB/s | 718 kB 00:00 2023-12-02T02:44:55.491 INFO:teuthology.orchestra.run.smithi057.stdout:(28/29): librados2-18.0.0-7631.gbed76814.el9.x8 1.3 MB/s | 3.3 MB 00:02 2023-12-02T02:44:55.497 INFO:teuthology.orchestra.run.smithi029.stdout:(4/29): ceph-radosgw-18.0.0-7631.gbed76814.el9. 2.4 MB/s | 8.6 MB 00:03 2023-12-02T02:44:55.530 INFO:teuthology.orchestra.run.smithi029.stdout:(5/29): libradosstriper1-18.0.0-7631.gbed76814. 1.7 MB/s | 475 kB 00:00 2023-12-02T02:44:55.639 INFO:teuthology.orchestra.run.smithi029.stdout:(6/29): python3-ceph-argparse-18.0.0-7631.gbed7 415 kB/s | 45 kB 00:00 2023-12-02T02:44:55.748 INFO:teuthology.orchestra.run.smithi029.stdout:(7/29): python3-ceph-common-18.0.0-7631.gbed768 1.2 MB/s | 129 kB 00:00 2023-12-02T02:44:55.867 INFO:teuthology.orchestra.run.smithi029.stdout:(8/29): python3-cephfs-18.0.0-7631.gbed76814.el 1.3 MB/s | 162 kB 00:00 2023-12-02T02:44:55.959 INFO:teuthology.orchestra.run.smithi029.stdout:(9/29): python3-rados-18.0.0-7631.gbed76814.el9 3.4 MB/s | 320 kB 00:00 2023-12-02T02:44:56.060 INFO:teuthology.orchestra.run.smithi029.stdout:(10/29): python3-rbd-18.0.0-7631.gbed76814.el9. 2.9 MB/s | 298 kB 00:00 2023-12-02T02:44:56.169 INFO:teuthology.orchestra.run.smithi029.stdout:(11/29): librgw2-18.0.0-7631.gbed76814.el9.x86_ 7.1 MB/s | 4.8 MB 00:00 2023-12-02T02:44:56.194 INFO:teuthology.orchestra.run.smithi029.stdout:(12/29): python3-rgw-18.0.0-7631.gbed76814.el9. 740 kB/s | 99 kB 00:00 2023-12-02T02:44:56.320 INFO:teuthology.orchestra.run.smithi029.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 704 kB/s | 106 kB 00:00 2023-12-02T02:44:56.345 INFO:teuthology.orchestra.run.smithi029.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2023-12-02T02:44:56.371 INFO:teuthology.orchestra.run.smithi029.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 872 kB/s | 44 kB 00:00 2023-12-02T02:44:56.421 INFO:teuthology.orchestra.run.smithi029.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 8.5 MB/s | 661 kB 00:00 2023-12-02T02:44:56.455 INFO:teuthology.orchestra.run.smithi029.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2023-12-02T02:44:56.466 INFO:teuthology.orchestra.run.smithi057.stdout:(29/29): ceph-common-18.0.0-7631.gbed76814.el9. 1.5 MB/s | 19 MB 00:12 2023-12-02T02:44:56.470 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:44:56.470 INFO:teuthology.orchestra.run.smithi057.stdout:Total 4.4 MB/s | 55 MB 00:12 2023-12-02T02:44:56.471 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2023-12-02T02:44:56.480 INFO:teuthology.orchestra.run.smithi029.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 702 kB/s | 41 kB 00:00 2023-12-02T02:44:56.510 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2023-12-02T02:44:56.510 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2023-12-02T02:44:56.898 INFO:teuthology.orchestra.run.smithi029.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 698 kB/s | 309 kB 00:00 2023-12-02T02:44:56.973 INFO:teuthology.orchestra.run.smithi029.stdout:(20/29): libarrow-doc-9.0.0-9.el9.noarch.rpm 324 kB/s | 24 kB 00:00 2023-12-02T02:44:56.984 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2023-12-02T02:44:56.985 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2023-12-02T02:44:57.049 INFO:teuthology.orchestra.run.smithi029.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 647 kB/s | 49 kB 00:00 2023-12-02T02:44:57.124 INFO:teuthology.orchestra.run.smithi029.stdout:(22/29): libunwind-1.6.2-1.el9.x86_64.rpm 897 kB/s | 67 kB 00:00 2023-12-02T02:44:57.235 INFO:teuthology.orchestra.run.smithi029.stdout:(23/29): libarrow-9.0.0-9.el9.x86_64.rpm 5.9 MB/s | 4.4 MB 00:00 2023-12-02T02:44:57.311 INFO:teuthology.orchestra.run.smithi029.stdout:(24/29): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2023-12-02T02:44:57.353 INFO:teuthology.orchestra.run.smithi029.stdout:(25/29): parquet-libs-9.0.0-9.el9.x86_64.rpm 3.6 MB/s | 837 kB 00:00 2023-12-02T02:44:57.428 INFO:teuthology.orchestra.run.smithi029.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2023-12-02T02:44:57.846 INFO:teuthology.orchestra.run.smithi029.stdout:(27/29): librbd1-18.0.0-7631.gbed76814.el9.x86_ 7.3 MB/s | 3.0 MB 00:00 2023-12-02T02:44:57.929 INFO:teuthology.orchestra.run.smithi029.stdout:(28/29): librados2-18.0.0-7631.gbed76814.el9.x8 5.8 MB/s | 3.3 MB 00:00 2023-12-02T02:44:58.499 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2023-12-02T02:44:58.646 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-02T02:44:58.680 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2023-12-02T02:44:58.737 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2023-12-02T02:44:59.091 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2023-12-02T02:44:59.159 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : librados2-2:18.0.0-7631.gbed76814.el9.x86_64 5/31 2023-12-02T02:44:59.251 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librados2-2:18.0.0-7631.gbed76814.el9.x86_64 5/31 2023-12-02T02:44:59.347 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rados-2:18.0.0-7631.gbed76814.el9.x86_64 6/31 2023-12-02T02:44:59.377 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libcephfs2-2:18.0.0-7631.gbed76814.el9.x86_64 7/31 2023-12-02T02:44:59.431 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libcephfs2-2:18.0.0-7631.gbed76814.el9.x86_64 7/31 2023-12-02T02:44:59.460 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2023-12-02T02:44:59.505 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-ceph-argparse-2:18.0.0-7631.gbed76814.el9. 9/31 2023-12-02T02:44:59.572 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cephfs-2:18.0.0-7631.gbed76814.el9.x86_64 10/31 2023-12-02T02:44:59.596 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libradosstriper1-2:18.0.0-7631.gbed76814.el9.x86_6 11/31 2023-12-02T02:44:59.645 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7631.gbed76814.el9.x86_6 11/31 2023-12-02T02:44:59.755 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-02T02:44:59.788 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/31 2023-12-02T02:44:59.829 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-02T02:44:59.860 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-02T02:44:59.890 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/31 2023-12-02T02:44:59.923 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-02T02:45:00.242 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-02T02:45:00.277 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : librbd1-2:18.0.0-7631.gbed76814.el9.x86_64 19/31 2023-12-02T02:45:00.346 INFO:teuthology.orchestra.run.smithi029.stdout:(29/29): ceph-common-18.0.0-7631.gbed76814.el9. 2.3 MB/s | 19 MB 00:08 2023-12-02T02:45:00.354 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:45:00.354 INFO:teuthology.orchestra.run.smithi029.stdout:Total 6.5 MB/s | 55 MB 00:08 2023-12-02T02:45:00.354 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-02T02:45:00.360 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librbd1-2:18.0.0-7631.gbed76814.el9.x86_64 19/31 2023-12-02T02:45:00.390 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rbd-2:18.0.0-7631.gbed76814.el9.x86_64 20/31 2023-12-02T02:45:00.395 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-02T02:45:00.396 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-02T02:45:00.426 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-02T02:45:00.879 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 22/31 2023-12-02T02:45:00.905 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librgw2-2:18.0.0-7631.gbed76814.el9.x86_64 23/31 2023-12-02T02:45:00.906 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-02T02:45:00.906 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-02T02:45:00.976 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librgw2-2:18.0.0-7631.gbed76814.el9.x86_64 23/31 2023-12-02T02:45:01.050 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rgw-2:18.0.0-7631.gbed76814.el9.x86_64 24/31 2023-12-02T02:45:01.076 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-ceph-common-2:18.0.0-7631.gbed76814.el9.x8 25/31 2023-12-02T02:45:02.287 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-02T02:45:02.382 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-02T02:45:02.420 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2023-12-02T02:45:02.490 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2023-12-02T02:45:02.841 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2023-12-02T02:45:02.977 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librados2-2:18.0.0-7631.gbed76814.el9.x86_64 5/31 2023-12-02T02:45:03.071 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:18.0.0-7631.gbed76814.el9.x86_64 5/31 2023-12-02T02:45:03.166 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rados-2:18.0.0-7631.gbed76814.el9.x86_64 6/31 2023-12-02T02:45:03.198 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs2-2:18.0.0-7631.gbed76814.el9.x86_64 7/31 2023-12-02T02:45:03.260 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs2-2:18.0.0-7631.gbed76814.el9.x86_64 7/31 2023-12-02T02:45:03.274 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-common-2:18.0.0-7631.gbed76814.el9.x86_64 26/31 2023-12-02T02:45:03.298 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2023-12-02T02:45:03.314 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-common-2:18.0.0-7631.gbed76814.el9.x86_64 26/31 2023-12-02T02:45:03.342 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-argparse-2:18.0.0-7631.gbed76814.el9. 9/31 2023-12-02T02:45:03.410 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cephfs-2:18.0.0-7631.gbed76814.el9.x86_64 10/31 2023-12-02T02:45:03.442 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libradosstriper1-2:18.0.0-7631.gbed76814.el9.x86_6 11/31 2023-12-02T02:45:03.499 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7631.gbed76814.el9.x86_6 11/31 2023-12-02T02:45:03.615 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-02T02:45:03.652 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/31 2023-12-02T02:45:03.694 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-02T02:45:03.736 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-02T02:45:03.765 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/31 2023-12-02T02:45:03.798 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-02T02:45:03.895 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-common-2:18.0.0-7631.gbed76814.el9.x86_64 26/31 2023-12-02T02:45:03.943 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-base-2:18.0.0-7631.gbed76814.el9.x86_64 27/31 2023-12-02T02:45:04.030 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-base-2:18.0.0-7631.gbed76814.el9.x86_64 27/31 2023-12-02T02:45:04.030 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-02T02:45:04.030 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-02T02:45:04.031 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:04.061 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-selinux-2:18.0.0-7631.gbed76814.el9.x86_64 28/31 2023-12-02T02:45:04.117 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-02T02:45:04.156 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librbd1-2:18.0.0-7631.gbed76814.el9.x86_64 19/31 2023-12-02T02:45:04.248 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:18.0.0-7631.gbed76814.el9.x86_64 19/31 2023-12-02T02:45:04.286 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rbd-2:18.0.0-7631.gbed76814.el9.x86_64 20/31 2023-12-02T02:45:04.332 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-02T02:45:04.782 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 22/31 2023-12-02T02:45:04.819 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librgw2-2:18.0.0-7631.gbed76814.el9.x86_64 23/31 2023-12-02T02:45:04.891 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librgw2-2:18.0.0-7631.gbed76814.el9.x86_64 23/31 2023-12-02T02:45:04.972 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rgw-2:18.0.0-7631.gbed76814.el9.x86_64 24/31 2023-12-02T02:45:05.008 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-common-2:18.0.0-7631.gbed76814.el9.x8 25/31 2023-12-02T02:45:07.243 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:18.0.0-7631.gbed76814.el9.x86_64 26/31 2023-12-02T02:45:07.295 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-common-2:18.0.0-7631.gbed76814.el9.x86_64 26/31 2023-12-02T02:45:07.866 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:18.0.0-7631.gbed76814.el9.x86_64 26/31 2023-12-02T02:45:07.919 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-base-2:18.0.0-7631.gbed76814.el9.x86_64 27/31 2023-12-02T02:45:08.020 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-base-2:18.0.0-7631.gbed76814.el9.x86_64 27/31 2023-12-02T02:45:08.020 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-02T02:45:08.021 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-02T02:45:08.021 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:45:08.063 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-selinux-2:18.0.0-7631.gbed76814.el9.x86_64 28/31 2023-12-02T02:45:13.946 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7631.gbed76814.el9.x86_64 28/31 2023-12-02T02:45:13.946 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /sys 2023-12-02T02:45:13.946 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /proc 2023-12-02T02:45:13.946 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /mnt 2023-12-02T02:45:13.946 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /var/tmp 2023-12-02T02:45:13.946 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /home 2023-12-02T02:45:13.946 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /root 2023-12-02T02:45:13.946 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /tmp 2023-12-02T02:45:13.947 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:14.781 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-radosgw-2:18.0.0-7631.gbed76814.el9.x86_64 29/31 2023-12-02T02:45:14.804 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7631.gbed76814.el9.x86_64 29/31 2023-12-02T02:45:14.804 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T02:45:14.804 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-02T02:45:14.804 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-02T02:45:14.804 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-02T02:45:14.804 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:14.806 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-02T02:45:14.892 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-02T02:45:14.892 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-02T02:45:16.040 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-02T02:45:16.040 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-base-2:18.0.0-7631.gbed76814.el9.x86_64 1/31 2023-12-02T02:45:16.040 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-common-2:18.0.0-7631.gbed76814.el9.x86_64 2/31 2023-12-02T02:45:16.040 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-radosgw-2:18.0.0-7631.gbed76814.el9.x86_64 3/31 2023-12-02T02:45:16.040 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-selinux-2:18.0.0-7631.gbed76814.el9.x86_64 4/31 2023-12-02T02:45:16.040 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libcephfs2-2:18.0.0-7631.gbed76814.el9.x86_64 5/31 2023-12-02T02:45:16.040 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libradosstriper1-2:18.0.0-7631.gbed76814.el9.x86_6 6/31 2023-12-02T02:45:16.040 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librgw2-2:18.0.0-7631.gbed76814.el9.x86_64 7/31 2023-12-02T02:45:16.040 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7631.gbed76814.el9. 8/31 2023-12-02T02:45:16.041 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-ceph-common-2:18.0.0-7631.gbed76814.el9.x8 9/31 2023-12-02T02:45:16.041 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cephfs-2:18.0.0-7631.gbed76814.el9.x86_64 10/31 2023-12-02T02:45:16.041 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rados-2:18.0.0-7631.gbed76814.el9.x86_64 11/31 2023-12-02T02:45:16.041 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rbd-2:18.0.0-7631.gbed76814.el9.x86_64 12/31 2023-12-02T02:45:16.041 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rgw-2:18.0.0-7631.gbed76814.el9.x86_64 13/31 2023-12-02T02:45:16.041 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-02T02:45:16.041 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-02T02:45:16.041 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-02T02:45:16.041 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-02T02:45:16.041 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-02T02:45:16.041 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-02T02:45:16.041 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-02T02:45:16.041 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 21/31 2023-12-02T02:45:16.041 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 22/31 2023-12-02T02:45:16.042 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-02T02:45:16.042 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-02T02:45:16.042 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 25/31 2023-12-02T02:45:16.042 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-02T02:45:16.042 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-02T02:45:16.042 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librados2-2:18.0.0-7631.gbed76814.el9.x86_64 28/31 2023-12-02T02:45:16.042 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-02T02:45:16.042 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librbd1-2:18.0.0-7631.gbed76814.el9.x86_64 30/31 2023-12-02T02:45:17.178 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-02T02:45:17.178 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:17.179 INFO:teuthology.orchestra.run.smithi057.stdout:Upgraded: 2023-12-02T02:45:17.179 INFO:teuthology.orchestra.run.smithi057.stdout: librados2-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:17.179 INFO:teuthology.orchestra.run.smithi057.stdout: librbd1-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:17.179 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2023-12-02T02:45:17.179 INFO:teuthology.orchestra.run.smithi057.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-02T02:45:17.179 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-base-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:17.180 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-common-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:17.180 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-radosgw-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:17.180 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-selinux-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:17.180 INFO:teuthology.orchestra.run.smithi057.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-02T02:45:17.180 INFO:teuthology.orchestra.run.smithi057.stdout: libarrow-9.0.0-9.el9.x86_64 2023-12-02T02:45:17.180 INFO:teuthology.orchestra.run.smithi057.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-12-02T02:45:17.180 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs2-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:17.180 INFO:teuthology.orchestra.run.smithi057.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-02T02:45:17.180 INFO:teuthology.orchestra.run.smithi057.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-02T02:45:17.180 INFO:teuthology.orchestra.run.smithi057.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-02T02:45:17.180 INFO:teuthology.orchestra.run.smithi057.stdout: libradosstriper1-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:17.180 INFO:teuthology.orchestra.run.smithi057.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-02T02:45:17.180 INFO:teuthology.orchestra.run.smithi057.stdout: librgw2-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:17.181 INFO:teuthology.orchestra.run.smithi057.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-02T02:45:17.181 INFO:teuthology.orchestra.run.smithi057.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-02T02:45:17.181 INFO:teuthology.orchestra.run.smithi057.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-12-02T02:45:17.181 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-argparse-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:17.181 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-common-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:17.181 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cephfs-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:17.181 INFO:teuthology.orchestra.run.smithi057.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-02T02:45:17.181 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rados-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:17.181 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rbd-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:17.181 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rgw-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:17.181 INFO:teuthology.orchestra.run.smithi057.stdout: re2-1:20211101-3.el9.x86_64 2023-12-02T02:45:17.181 INFO:teuthology.orchestra.run.smithi057.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-02T02:45:17.181 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:17.181 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-12-02T02:45:17.577 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-test 2023-12-02T02:45:18.184 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:00:37 ago on Sat 02 Dec 2023 02:44:41 AM UTC. 2023-12-02T02:45:18.344 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-12-02T02:45:18.344 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:45:18.344 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2023-12-02T02:45:18.345 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:45:18.345 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2023-12-02T02:45:18.345 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-test x86_64 2:18.0.0-7631.gbed76814.el9 ceph 41 M 2023-12-02T02:45:18.345 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2023-12-02T02:45:18.345 INFO:teuthology.orchestra.run.smithi057.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-02T02:45:18.345 INFO:teuthology.orchestra.run.smithi057.stdout: libcephsqlite x86_64 2:18.0.0-7631.gbed76814.el9 ceph 164 k 2023-12-02T02:45:18.345 INFO:teuthology.orchestra.run.smithi057.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-02T02:45:18.345 INFO:teuthology.orchestra.run.smithi057.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-02T02:45:18.345 INFO:teuthology.orchestra.run.smithi057.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-02T02:45:18.345 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:18.345 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2023-12-02T02:45:18.345 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:45:18.345 INFO:teuthology.orchestra.run.smithi057.stdout:Install 6 Packages 2023-12-02T02:45:18.346 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:18.346 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 42 M 2023-12-02T02:45:18.346 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 173 M 2023-12-02T02:45:18.346 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2023-12-02T02:45:18.496 INFO:teuthology.orchestra.run.smithi057.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 186 kB 00:00 2023-12-02T02:45:18.547 INFO:teuthology.orchestra.run.smithi057.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2023-12-02T02:45:18.598 INFO:teuthology.orchestra.run.smithi057.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2023-12-02T02:45:18.622 INFO:teuthology.orchestra.run.smithi057.stdout:(4/6): libcephsqlite-18.0.0-7631.gbed76814.el9. 595 kB/s | 164 kB 00:00 2023-12-02T02:45:18.639 INFO:teuthology.orchestra.run.smithi057.stdout:(5/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2023-12-02T02:45:19.928 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7631.gbed76814.el9.x86_64 28/31 2023-12-02T02:45:19.928 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /sys 2023-12-02T02:45:19.928 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /proc 2023-12-02T02:45:19.928 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /mnt 2023-12-02T02:45:19.928 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /var/tmp 2023-12-02T02:45:19.928 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /home 2023-12-02T02:45:19.928 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /root 2023-12-02T02:45:19.929 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /tmp 2023-12-02T02:45:19.929 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:45:20.714 INFO:teuthology.orchestra.run.smithi057.stdout:(6/6): ceph-test-18.0.0-7631.gbed76814.el9.x86_ 17 MB/s | 41 MB 00:02 2023-12-02T02:45:20.717 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:45:20.718 INFO:teuthology.orchestra.run.smithi057.stdout:Total 18 MB/s | 42 MB 00:02 2023-12-02T02:45:20.718 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2023-12-02T02:45:20.740 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2023-12-02T02:45:20.741 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2023-12-02T02:45:20.794 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-radosgw-2:18.0.0-7631.gbed76814.el9.x86_64 29/31 2023-12-02T02:45:20.825 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7631.gbed76814.el9.x86_64 29/31 2023-12-02T02:45:20.826 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T02:45:20.826 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-02T02:45:20.826 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-02T02:45:20.826 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-02T02:45:20.826 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:45:20.829 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-02T02:45:20.924 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-02T02:45:20.924 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-02T02:45:21.107 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2023-12-02T02:45:21.107 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2023-12-02T02:45:21.662 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2023-12-02T02:45:21.741 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-02T02:45:21.783 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-02T02:45:21.814 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-02T02:45:21.866 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-02T02:45:21.892 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libcephsqlite-2:18.0.0-7631.gbed76814.el9.x86_64 5/6 2023-12-02T02:45:22.342 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-02T02:45:22.342 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-base-2:18.0.0-7631.gbed76814.el9.x86_64 1/31 2023-12-02T02:45:22.342 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-common-2:18.0.0-7631.gbed76814.el9.x86_64 2/31 2023-12-02T02:45:22.342 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-radosgw-2:18.0.0-7631.gbed76814.el9.x86_64 3/31 2023-12-02T02:45:22.342 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-selinux-2:18.0.0-7631.gbed76814.el9.x86_64 4/31 2023-12-02T02:45:22.343 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs2-2:18.0.0-7631.gbed76814.el9.x86_64 5/31 2023-12-02T02:45:22.343 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libradosstriper1-2:18.0.0-7631.gbed76814.el9.x86_6 6/31 2023-12-02T02:45:22.343 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librgw2-2:18.0.0-7631.gbed76814.el9.x86_64 7/31 2023-12-02T02:45:22.343 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7631.gbed76814.el9. 8/31 2023-12-02T02:45:22.343 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-common-2:18.0.0-7631.gbed76814.el9.x8 9/31 2023-12-02T02:45:22.343 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cephfs-2:18.0.0-7631.gbed76814.el9.x86_64 10/31 2023-12-02T02:45:22.343 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rados-2:18.0.0-7631.gbed76814.el9.x86_64 11/31 2023-12-02T02:45:22.343 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rbd-2:18.0.0-7631.gbed76814.el9.x86_64 12/31 2023-12-02T02:45:22.343 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rgw-2:18.0.0-7631.gbed76814.el9.x86_64 13/31 2023-12-02T02:45:22.343 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-02T02:45:22.343 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-02T02:45:22.343 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-02T02:45:22.343 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-02T02:45:22.344 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-02T02:45:22.344 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-02T02:45:22.344 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-02T02:45:22.344 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 21/31 2023-12-02T02:45:22.344 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 22/31 2023-12-02T02:45:22.344 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-02T02:45:22.344 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-02T02:45:22.344 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 25/31 2023-12-02T02:45:22.344 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-02T02:45:22.344 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-02T02:45:22.344 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:18.0.0-7631.gbed76814.el9.x86_64 28/31 2023-12-02T02:45:22.344 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-02T02:45:22.345 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:18.0.0-7631.gbed76814.el9.x86_64 30/31 2023-12-02T02:45:23.319 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-02T02:45:23.319 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:45:23.319 INFO:teuthology.orchestra.run.smithi029.stdout:Upgraded: 2023-12-02T02:45:23.320 INFO:teuthology.orchestra.run.smithi029.stdout: librados2-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:23.320 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:23.320 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-02T02:45:23.320 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-02T02:45:23.320 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:23.320 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:23.320 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:23.321 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:23.321 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-02T02:45:23.321 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-9.0.0-9.el9.x86_64 2023-12-02T02:45:23.321 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-12-02T02:45:23.321 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:23.321 INFO:teuthology.orchestra.run.smithi029.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-02T02:45:23.321 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-02T02:45:23.321 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-02T02:45:23.321 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:23.321 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-02T02:45:23.321 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:23.321 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-02T02:45:23.322 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-02T02:45:23.322 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-12-02T02:45:23.322 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:23.322 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:23.322 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:23.322 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-02T02:45:23.322 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:23.322 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:23.322 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:23.322 INFO:teuthology.orchestra.run.smithi029.stdout: re2-1:20211101-3.el9.x86_64 2023-12-02T02:45:23.322 INFO:teuthology.orchestra.run.smithi029.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-02T02:45:23.322 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:45:23.323 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-02T02:45:23.752 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-test 2023-12-02T02:45:24.354 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:00:36 ago on Sat 02 Dec 2023 02:44:48 AM UTC. 2023-12-02T02:45:24.511 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-02T02:45:24.512 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:45:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-12-02T02:45:24.512 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:45:24.512 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-02T02:45:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test x86_64 2:18.0.0-7631.gbed76814.el9 ceph 41 M 2023-12-02T02:45:24.512 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-12-02T02:45:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-02T02:45:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite x86_64 2:18.0.0-7631.gbed76814.el9 ceph 164 k 2023-12-02T02:45:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-02T02:45:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-02T02:45:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-02T02:45:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:45:24.513 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-02T02:45:24.513 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:45:24.513 INFO:teuthology.orchestra.run.smithi029.stdout:Install 6 Packages 2023-12-02T02:45:24.513 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:45:24.513 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 42 M 2023-12-02T02:45:24.513 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 173 M 2023-12-02T02:45:24.513 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-02T02:45:24.689 INFO:teuthology.orchestra.run.smithi029.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 186 kB 00:00 2023-12-02T02:45:24.739 INFO:teuthology.orchestra.run.smithi029.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2023-12-02T02:45:24.782 INFO:teuthology.orchestra.run.smithi029.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2023-12-02T02:45:24.825 INFO:teuthology.orchestra.run.smithi029.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2023-12-02T02:45:25.149 INFO:teuthology.orchestra.run.smithi029.stdout:(5/6): libcephsqlite-18.0.0-7631.gbed76814.el9. 258 kB/s | 164 kB 00:00 2023-12-02T02:45:25.975 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7631.gbed76814.el9.x86_64 5/6 2023-12-02T02:45:26.052 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-test-2:18.0.0-7631.gbed76814.el9.x86_64 6/6 2023-12-02T02:45:28.134 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-test-2:18.0.0-7631.gbed76814.el9.x86_64 6/6 2023-12-02T02:45:28.134 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-test-2:18.0.0-7631.gbed76814.el9.x86_64 1/6 2023-12-02T02:45:28.134 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libcephsqlite-2:18.0.0-7631.gbed76814.el9.x86_64 2/6 2023-12-02T02:45:28.134 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-02T02:45:28.134 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-02T02:45:28.134 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-02T02:45:28.554 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-02T02:45:28.554 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:28.554 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2023-12-02T02:45:28.554 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-test-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:28.554 INFO:teuthology.orchestra.run.smithi057.stdout: jq-1.6-16.el9.x86_64 2023-12-02T02:45:28.554 INFO:teuthology.orchestra.run.smithi057.stdout: libcephsqlite-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:28.554 INFO:teuthology.orchestra.run.smithi057.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-02T02:45:28.554 INFO:teuthology.orchestra.run.smithi057.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-02T02:45:28.555 INFO:teuthology.orchestra.run.smithi057.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-02T02:45:28.555 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:28.555 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-12-02T02:45:28.865 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph 2023-12-02T02:45:29.466 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:00:48 ago on Sat 02 Dec 2023 02:44:41 AM UTC. 2023-12-02T02:45:29.622 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-12-02T02:45:29.624 INFO:teuthology.orchestra.run.smithi057.stdout:========================================================================================= 2023-12-02T02:45:29.624 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2023-12-02T02:45:29.625 INFO:teuthology.orchestra.run.smithi057.stdout:========================================================================================= 2023-12-02T02:45:29.625 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2023-12-02T02:45:29.625 INFO:teuthology.orchestra.run.smithi057.stdout: ceph x86_64 2:18.0.0-7631.gbed76814.el9 ceph 6.6 k 2023-12-02T02:45:29.625 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2023-12-02T02:45:29.625 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mds x86_64 2:18.0.0-7631.gbed76814.el9 ceph 2.1 M 2023-12-02T02:45:29.625 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr x86_64 2:18.0.0-7631.gbed76814.el9 ceph 1.5 M 2023-12-02T02:45:29.625 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7631.gbed76814.el9 ceph-noarch 243 k 2023-12-02T02:45:29.625 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mon x86_64 2:18.0.0-7631.gbed76814.el9 ceph 4.4 M 2023-12-02T02:45:29.625 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-osd x86_64 2:18.0.0-7631.gbed76814.el9 ceph 18 M 2023-12-02T02:45:29.625 INFO:teuthology.orchestra.run.smithi057.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-02T02:45:29.625 INFO:teuthology.orchestra.run.smithi057.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-12-02T02:45:29.625 INFO:teuthology.orchestra.run.smithi057.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-02T02:45:29.625 INFO:teuthology.orchestra.run.smithi057.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-02T02:45:29.626 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-02T02:45:29.626 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-02T02:45:29.626 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-02T02:45:29.626 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-02T02:45:29.626 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-02T02:45:29.626 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-02T02:45:29.626 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-02T02:45:29.626 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-02T02:45:29.626 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-02T02:45:29.626 INFO:teuthology.orchestra.run.smithi057.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-02T02:45:29.626 INFO:teuthology.orchestra.run.smithi057.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-02T02:45:29.626 INFO:teuthology.orchestra.run.smithi057.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-02T02:45:29.626 INFO:teuthology.orchestra.run.smithi057.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-02T02:45:29.626 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-02T02:45:29.627 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-02T02:45:29.627 INFO:teuthology.orchestra.run.smithi057.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-02T02:45:29.627 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-02T02:45:29.627 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-02T02:45:29.627 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-02T02:45:29.627 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-02T02:45:29.627 INFO:teuthology.orchestra.run.smithi057.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-02T02:45:29.627 INFO:teuthology.orchestra.run.smithi057.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-02T02:45:29.627 INFO:teuthology.orchestra.run.smithi057.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-02T02:45:29.627 INFO:teuthology.orchestra.run.smithi057.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-02T02:45:29.627 INFO:teuthology.orchestra.run.smithi057.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-02T02:45:29.627 INFO:teuthology.orchestra.run.smithi057.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-02T02:45:29.627 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:29.627 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2023-12-02T02:45:29.628 INFO:teuthology.orchestra.run.smithi057.stdout:========================================================================================= 2023-12-02T02:45:29.628 INFO:teuthology.orchestra.run.smithi057.stdout:Install 36 Packages 2023-12-02T02:45:29.628 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:29.628 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 30 M 2023-12-02T02:45:29.628 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 111 M 2023-12-02T02:45:29.628 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2023-12-02T02:45:30.330 INFO:teuthology.orchestra.run.smithi057.stdout:(1/36): ceph-18.0.0-7631.gbed76814.el9.x86_64.r 25 kB/s | 6.6 kB 00:00 2023-12-02T02:45:30.680 INFO:teuthology.orchestra.run.smithi057.stdout:(2/36): ceph-mon-18.0.0-7631.gbed76814.el9.x86_ 13 MB/s | 4.4 MB 00:00 2023-12-02T02:45:31.539 INFO:teuthology.orchestra.run.smithi057.stdout:(3/36): ceph-osd-18.0.0-7631.gbed76814.el9.x86_ 21 MB/s | 18 MB 00:00 2023-12-02T02:45:31.589 INFO:teuthology.orchestra.run.smithi057.stdout:(4/36): ceph-mgr-modules-core-18.0.0-7631.gbed7 4.8 MB/s | 243 kB 00:00 2023-12-02T02:45:31.622 INFO:teuthology.orchestra.run.smithi057.stdout:(5/36): ceph-mgr-18.0.0-7631.gbed76814.el9.x86_ 966 kB/s | 1.5 MB 00:01 2023-12-02T02:45:31.923 INFO:teuthology.orchestra.run.smithi057.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 560 kB/s | 187 kB 00:00 2023-12-02T02:45:31.948 INFO:teuthology.orchestra.run.smithi057.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 443 kB/s | 144 kB 00:00 2023-12-02T02:45:31.981 INFO:teuthology.orchestra.run.smithi057.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 541 kB/s | 32 kB 00:00 2023-12-02T02:45:32.007 INFO:teuthology.orchestra.run.smithi057.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 808 kB/s | 47 kB 00:00 2023-12-02T02:45:32.049 INFO:teuthology.orchestra.run.smithi057.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 575 kB/s | 39 kB 00:00 2023-12-02T02:45:32.107 INFO:teuthology.orchestra.run.smithi057.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.3 MB/s | 241 kB 00:00 2023-12-02T02:45:32.216 INFO:teuthology.orchestra.run.smithi057.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 6.8 MB/s | 1.1 MB 00:00 2023-12-02T02:45:32.241 INFO:teuthology.orchestra.run.smithi057.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 772 kB/s | 103 kB 00:00 2023-12-02T02:45:32.275 INFO:teuthology.orchestra.run.smithi057.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.1 MB/s | 124 kB 00:00 2023-12-02T02:45:32.309 INFO:teuthology.orchestra.run.smithi057.stdout:(15/36): python3-requests-2.25.1-7.el9.noarch.r 1.6 MB/s | 113 kB 00:00 2023-12-02T02:45:32.350 INFO:teuthology.orchestra.run.smithi057.stdout:(16/36): python3-urllib3-1.26.5-4.el9.noarch.rp 2.4 MB/s | 187 kB 00:00 2023-12-02T02:45:32.575 INFO:teuthology.orchestra.run.smithi057.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 672 kB/s | 151 kB 00:00 2023-12-02T02:45:32.634 INFO:teuthology.orchestra.run.smithi057.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 743 kB/s | 43 kB 00:00 2023-12-02T02:45:32.726 INFO:teuthology.orchestra.run.smithi057.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2023-12-02T02:45:32.835 INFO:teuthology.orchestra.run.smithi057.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.2 MB/s | 358 kB 00:00 2023-12-02T02:45:32.893 INFO:teuthology.orchestra.run.smithi057.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 183 kB/s | 11 kB 00:00 2023-12-02T02:45:32.952 INFO:teuthology.orchestra.run.smithi057.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 303 kB/s | 18 kB 00:00 2023-12-02T02:45:33.010 INFO:teuthology.orchestra.run.smithi057.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 397 kB/s | 23 kB 00:00 2023-12-02T02:45:33.036 INFO:teuthology.orchestra.run.smithi057.stdout:(24/36): lua-devel-5.4.4-4.el9.x86_64.rpm 31 kB/s | 22 kB 00:00 2023-12-02T02:45:33.078 INFO:teuthology.orchestra.run.smithi057.stdout:(25/36): ceph-mds-18.0.0-7631.gbed76814.el9.x86 710 kB/s | 2.1 MB 00:03 2023-12-02T02:45:33.119 INFO:teuthology.orchestra.run.smithi057.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 180 kB/s | 19 kB 00:00 2023-12-02T02:45:33.170 INFO:teuthology.orchestra.run.smithi057.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 147 kB/s | 20 kB 00:00 2023-12-02T02:45:33.195 INFO:teuthology.orchestra.run.smithi057.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2023-12-02T02:45:33.237 INFO:teuthology.orchestra.run.smithi057.stdout:(29/36): python3-logutils-0.3.5-21.el9.noarch.r 292 kB/s | 46 kB 00:00 2023-12-02T02:45:33.262 INFO:teuthology.orchestra.run.smithi057.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 246 kB/s | 16 kB 00:00 2023-12-02T02:45:33.321 INFO:teuthology.orchestra.run.smithi057.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 613 kB/s | 36 kB 00:00 2023-12-02T02:45:33.338 INFO:teuthology.orchestra.run.smithi057.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 895 kB/s | 90 kB 00:00 2023-12-02T02:45:33.363 INFO:teuthology.orchestra.run.smithi057.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2023-12-02T02:45:33.388 INFO:teuthology.orchestra.run.smithi057.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.3 MB/s | 230 kB 00:00 2023-12-02T02:45:33.413 INFO:teuthology.orchestra.run.smithi057.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 400 kB/s | 20 kB 00:00 2023-12-02T02:45:33.464 INFO:teuthology.orchestra.run.smithi057.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2023-12-02T02:45:33.468 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:45:33.468 INFO:teuthology.orchestra.run.smithi057.stdout:Total 7.9 MB/s | 30 MB 00:03 2023-12-02T02:45:33.481 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2023-12-02T02:45:33.529 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2023-12-02T02:45:33.529 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2023-12-02T02:45:33.945 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2023-12-02T02:45:33.946 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2023-12-02T02:45:35.300 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2023-12-02T02:45:35.358 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2023-12-02T02:45:35.392 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2023-12-02T02:45:35.419 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2023-12-02T02:45:35.485 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2023-12-02T02:45:35.526 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2023-12-02T02:45:35.559 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2023-12-02T02:45:35.654 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2023-12-02T02:45:35.732 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2023-12-02T02:45:35.779 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2023-12-02T02:45:35.853 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2023-12-02T02:45:35.905 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 11/36 2023-12-02T02:45:35.950 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2023-12-02T02:45:36.001 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2023-12-02T02:45:36.057 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2023-12-02T02:45:36.259 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2023-12-02T02:45:36.314 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2023-12-02T02:45:36.361 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2023-12-02T02:45:36.395 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2023-12-02T02:45:36.424 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2023-12-02T02:45:36.503 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2023-12-02T02:45:36.543 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-02T02:45:36.591 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2023-12-02T02:45:36.634 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2023-12-02T02:45:36.677 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2023-12-02T02:45:36.728 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2023-12-02T02:45:36.830 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2023-12-02T02:45:36.868 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2023-12-02T02:45:36.943 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2023-12-02T02:45:37.092 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2023-12-02T02:45:37.232 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-02T02:45:37.395 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7631.gbed76814.el9. 31/36 2023-12-02T02:45:37.421 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-2:18.0.0-7631.gbed76814.el9.x86_64 32/36 2023-12-02T02:45:37.447 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7631.gbed76814.el9.x86_64 32/36 2023-12-02T02:45:37.447 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T02:45:37.447 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-02T02:45:37.447 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-02T02:45:37.447 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-02T02:45:37.447 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:39.056 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-osd-2:18.0.0-7631.gbed76814.el9.x86_64 33/36 2023-12-02T02:45:39.086 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-osd-2:18.0.0-7631.gbed76814.el9.x86_64 33/36 2023-12-02T02:45:39.086 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T02:45:39.086 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-02T02:45:39.086 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-02T02:45:39.087 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-02T02:45:39.087 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:39.573 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mon-2:18.0.0-7631.gbed76814.el9.x86_64 34/36 2023-12-02T02:45:39.598 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mon-2:18.0.0-7631.gbed76814.el9.x86_64 34/36 2023-12-02T02:45:39.599 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T02:45:39.599 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-02T02:45:39.599 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-02T02:45:39.599 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-02T02:45:39.599 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:39.807 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mds-2:18.0.0-7631.gbed76814.el9.x86_64 35/36 2023-12-02T02:45:39.832 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mds-2:18.0.0-7631.gbed76814.el9.x86_64 35/36 2023-12-02T02:45:39.832 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T02:45:39.832 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-02T02:45:39.832 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-02T02:45:39.833 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-02T02:45:39.833 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:39.900 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-2:18.0.0-7631.gbed76814.el9.x86_64 36/36 2023-12-02T02:45:41.621 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-2:18.0.0-7631.gbed76814.el9.x86_64 36/36 2023-12-02T02:45:41.622 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-2:18.0.0-7631.gbed76814.el9.x86_64 1/36 2023-12-02T02:45:41.622 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mds-2:18.0.0-7631.gbed76814.el9.x86_64 2/36 2023-12-02T02:45:41.622 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-2:18.0.0-7631.gbed76814.el9.x86_64 3/36 2023-12-02T02:45:41.622 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mon-2:18.0.0-7631.gbed76814.el9.x86_64 4/36 2023-12-02T02:45:41.622 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-osd-2:18.0.0-7631.gbed76814.el9.x86_64 5/36 2023-12-02T02:45:41.622 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7631.gbed76814.el9. 6/36 2023-12-02T02:45:41.622 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2023-12-02T02:45:41.622 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2023-12-02T02:45:41.622 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2023-12-02T02:45:41.622 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2023-12-02T02:45:41.622 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2023-12-02T02:45:41.622 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2023-12-02T02:45:41.623 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2023-12-02T02:45:41.623 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2023-12-02T02:45:41.623 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2023-12-02T02:45:41.623 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2023-12-02T02:45:41.623 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/36 2023-12-02T02:45:41.623 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2023-12-02T02:45:41.623 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2023-12-02T02:45:41.623 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2023-12-02T02:45:41.623 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-02T02:45:41.623 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2023-12-02T02:45:41.623 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2023-12-02T02:45:41.623 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2023-12-02T02:45:41.624 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2023-12-02T02:45:41.624 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2023-12-02T02:45:41.624 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2023-12-02T02:45:41.625 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2023-12-02T02:45:41.625 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2023-12-02T02:45:41.626 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-02T02:45:41.626 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2023-12-02T02:45:41.626 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2023-12-02T02:45:41.626 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2023-12-02T02:45:41.626 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2023-12-02T02:45:41.626 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2023-12-02T02:45:42.567 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2023-12-02T02:45:42.568 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:42.568 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2023-12-02T02:45:42.568 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:42.568 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mds-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:42.568 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:42.568 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-modules-core-2:18.0.0-7631.gbed76814.el9.noarch 2023-12-02T02:45:42.568 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mon-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:42.568 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-osd-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:42.568 INFO:teuthology.orchestra.run.smithi057.stdout: lua-5.4.4-4.el9.x86_64 2023-12-02T02:45:42.568 INFO:teuthology.orchestra.run.smithi057.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-12-02T02:45:42.568 INFO:teuthology.orchestra.run.smithi057.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-02T02:45:42.568 INFO:teuthology.orchestra.run.smithi057.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-02T02:45:42.569 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-02T02:45:42.569 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-02T02:45:42.569 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-02T02:45:42.569 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-02T02:45:42.569 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-02T02:45:42.569 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-02T02:45:42.569 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-02T02:45:42.569 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-02T02:45:42.569 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-02T02:45:42.569 INFO:teuthology.orchestra.run.smithi057.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-02T02:45:42.569 INFO:teuthology.orchestra.run.smithi057.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-02T02:45:42.569 INFO:teuthology.orchestra.run.smithi057.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-02T02:45:42.569 INFO:teuthology.orchestra.run.smithi057.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-02T02:45:42.570 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-02T02:45:42.570 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ply-3.11-14.el9.noarch 2023-12-02T02:45:42.570 INFO:teuthology.orchestra.run.smithi057.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-02T02:45:42.570 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-02T02:45:42.570 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-02T02:45:42.570 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-02T02:45:42.570 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-02T02:45:42.570 INFO:teuthology.orchestra.run.smithi057.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-02T02:45:42.570 INFO:teuthology.orchestra.run.smithi057.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-02T02:45:42.570 INFO:teuthology.orchestra.run.smithi057.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-02T02:45:42.570 INFO:teuthology.orchestra.run.smithi057.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-02T02:45:42.570 INFO:teuthology.orchestra.run.smithi057.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-02T02:45:42.570 INFO:teuthology.orchestra.run.smithi057.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-02T02:45:42.570 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:42.570 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-12-02T02:45:42.902 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-base 2023-12-02T02:45:43.307 INFO:teuthology.orchestra.run.smithi029.stdout:(6/6): ceph-test-18.0.0-7631.gbed76814.el9.x86_ 2.2 MB/s | 41 MB 00:18 2023-12-02T02:45:43.310 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:45:43.311 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.2 MB/s | 42 MB 00:18 2023-12-02T02:45:43.311 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-02T02:45:43.333 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-02T02:45:43.333 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-02T02:45:43.501 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:02 ago on Sat 02 Dec 2023 02:44:41 AM UTC. 2023-12-02T02:45:43.600 INFO:teuthology.orchestra.run.smithi057.stdout:Package ceph-base-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T02:45:43.652 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-12-02T02:45:43.653 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2023-12-02T02:45:43.653 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-12-02T02:45:43.699 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-02T02:45:43.699 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-02T02:45:43.705 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install cephadm 2023-12-02T02:45:44.298 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:03 ago on Sat 02 Dec 2023 02:44:41 AM UTC. 2023-12-02T02:45:44.299 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-02T02:45:44.377 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-02T02:45:44.427 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-02T02:45:44.449 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-12-02T02:45:44.450 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:45:44.450 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2023-12-02T02:45:44.450 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:45:44.450 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2023-12-02T02:45:44.450 INFO:teuthology.orchestra.run.smithi057.stdout: cephadm noarch 2:18.0.0-7631.gbed76814.el9 ceph-noarch 776 k 2023-12-02T02:45:44.450 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:44.450 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2023-12-02T02:45:44.450 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:45:44.450 INFO:teuthology.orchestra.run.smithi057.stdout:Install 1 Package 2023-12-02T02:45:44.450 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:44.451 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 776 k 2023-12-02T02:45:44.451 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 790 k 2023-12-02T02:45:44.451 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2023-12-02T02:45:44.469 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-02T02:45:44.512 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-02T02:45:44.540 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephsqlite-2:18.0.0-7631.gbed76814.el9.x86_64 5/6 2023-12-02T02:45:44.730 INFO:teuthology.orchestra.run.smithi057.stdout:cephadm-18.0.0-7631.gbed76814.el9.noarch.rpm 2.7 MB/s | 776 kB 00:00 2023-12-02T02:45:44.730 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:45:44.731 INFO:teuthology.orchestra.run.smithi057.stdout:Total 2.7 MB/s | 776 kB 00:00 2023-12-02T02:45:44.731 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2023-12-02T02:45:44.736 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2023-12-02T02:45:44.737 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2023-12-02T02:45:44.743 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2023-12-02T02:45:44.743 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2023-12-02T02:45:44.816 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2023-12-02T02:45:45.309 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: cephadm-2:18.0.0-7631.gbed76814.el9.noarch 1/1 2023-12-02T02:45:45.403 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : cephadm-2:18.0.0-7631.gbed76814.el9.noarch 1/1 2023-12-02T02:45:45.787 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: cephadm-2:18.0.0-7631.gbed76814.el9.noarch 1/1 2023-12-02T02:45:46.051 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : cephadm-2:18.0.0-7631.gbed76814.el9.noarch 1/1 2023-12-02T02:45:46.051 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:46.051 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2023-12-02T02:45:46.051 INFO:teuthology.orchestra.run.smithi057.stdout: cephadm-2:18.0.0-7631.gbed76814.el9.noarch 2023-12-02T02:45:46.051 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:46.051 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-12-02T02:45:46.261 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-immutable-object-cache 2023-12-02T02:45:46.874 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:05 ago on Sat 02 Dec 2023 02:44:41 AM UTC. 2023-12-02T02:45:47.025 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-12-02T02:45:47.026 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:45:47.026 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repo Size 2023-12-02T02:45:47.026 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:45:47.026 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2023-12-02T02:45:47.026 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7631.gbed76814.el9 ceph 141 k 2023-12-02T02:45:47.026 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:47.027 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2023-12-02T02:45:47.027 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:45:47.027 INFO:teuthology.orchestra.run.smithi057.stdout:Install 1 Package 2023-12-02T02:45:47.027 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:47.027 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 141 k 2023-12-02T02:45:47.027 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 400 k 2023-12-02T02:45:47.027 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2023-12-02T02:45:47.260 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-immutable-object-cache-18.0.0-7631.gbed768 606 kB/s | 141 kB 00:00 2023-12-02T02:45:47.260 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:45:47.261 INFO:teuthology.orchestra.run.smithi057.stdout:Total 602 kB/s | 141 kB 00:00 2023-12-02T02:45:47.261 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2023-12-02T02:45:47.269 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2023-12-02T02:45:47.269 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2023-12-02T02:45:47.324 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2023-12-02T02:45:47.324 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2023-12-02T02:45:47.466 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2023-12-02T02:45:47.527 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7631.gbed76814. 1/1 2023-12-02T02:45:47.553 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7631.gbed76814. 1/1 2023-12-02T02:45:47.553 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T02:45:47.554 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-02T02:45:47.554 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:48.558 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7631.gbed76814. 1/1 2023-12-02T02:45:48.558 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:48.559 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2023-12-02T02:45:48.559 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-immutable-object-cache-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:48.559 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:48.559 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-12-02T02:45:48.677 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7631.gbed76814.el9.x86_64 5/6 2023-12-02T02:45:48.764 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-test-2:18.0.0-7631.gbed76814.el9.x86_64 6/6 2023-12-02T02:45:48.777 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-mgr 2023-12-02T02:45:49.382 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:08 ago on Sat 02 Dec 2023 02:44:41 AM UTC. 2023-12-02T02:45:49.483 INFO:teuthology.orchestra.run.smithi057.stdout:Package ceph-mgr-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T02:45:49.537 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-12-02T02:45:49.538 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2023-12-02T02:45:49.538 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-12-02T02:45:49.595 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-mgr-dashboard 2023-12-02T02:45:50.203 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:09 ago on Sat 02 Dec 2023 02:44:41 AM UTC. 2023-12-02T02:45:50.356 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-12-02T02:45:50.357 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:45:50.357 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2023-12-02T02:45:50.357 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:45:50.357 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2023-12-02T02:45:50.357 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7631.gbed76814.el9 ceph-noarch 1.8 M 2023-12-02T02:45:50.357 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2023-12-02T02:45:50.357 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7631.gbed76814.el9 ceph-noarch 28 k 2023-12-02T02:45:50.357 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7631.gbed76814.el9 ceph-noarch 15 k 2023-12-02T02:45:50.357 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-02T02:45:50.357 INFO:teuthology.orchestra.run.smithi057.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-02T02:45:50.358 INFO:teuthology.orchestra.run.smithi057.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-02T02:45:50.358 INFO:teuthology.orchestra.run.smithi057.stdout:Installing weak dependencies: 2023-12-02T02:45:50.358 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-02T02:45:50.358 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:50.358 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2023-12-02T02:45:50.358 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:45:50.358 INFO:teuthology.orchestra.run.smithi057.stdout:Install 7 Packages 2023-12-02T02:45:50.358 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:50.359 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 2.1 M 2023-12-02T02:45:50.359 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 8.7 M 2023-12-02T02:45:50.359 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2023-12-02T02:45:50.605 INFO:teuthology.orchestra.run.smithi057.stdout:(1/7): ceph-grafana-dashboards-18.0.0-7631.gbed 152 kB/s | 28 kB 00:00 2023-12-02T02:45:50.697 INFO:teuthology.orchestra.run.smithi057.stdout:(2/7): ceph-prometheus-alerts-18.0.0-7631.gbed7 55 kB/s | 15 kB 00:00 2023-12-02T02:45:50.763 INFO:teuthology.orchestra.run.smithi057.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 57 kB/s | 9.0 kB 00:00 2023-12-02T02:45:50.805 INFO:teuthology.orchestra.run.smithi057.stdout:(4/7): ceph-mgr-dashboard-18.0.0-7631.gbed76814 4.6 MB/s | 1.8 MB 00:00 2023-12-02T02:45:50.831 INFO:teuthology.orchestra.run.smithi057.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 467 kB/s | 31 kB 00:00 2023-12-02T02:45:50.881 INFO:teuthology.orchestra.run.smithi057.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 222 kB/s | 41 kB 00:00 2023-12-02T02:45:51.006 INFO:teuthology.orchestra.run.smithi057.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 940 kB/s | 188 kB 00:00 2023-12-02T02:45:51.011 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:45:51.011 INFO:teuthology.orchestra.run.smithi057.stdout:Total 3.2 MB/s | 2.1 MB 00:00 2023-12-02T02:45:51.011 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2023-12-02T02:45:51.019 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2023-12-02T02:45:51.019 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2023-12-02T02:45:51.094 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2023-12-02T02:45:51.094 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2023-12-02T02:45:51.397 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2023-12-02T02:45:51.466 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-02T02:45:51.501 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-02T02:45:51.547 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-02T02:45:51.578 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-02T02:45:51.609 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7631.gbed76814.el9.n 5/7 2023-12-02T02:45:51.857 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7631.gbed76814.el9. 6/7 2023-12-02T02:45:51.893 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7631.gbed76814.el9.noarc 7/7 2023-12-02T02:45:52.094 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-test-2:18.0.0-7631.gbed76814.el9.x86_64 6/6 2023-12-02T02:45:52.094 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-test-2:18.0.0-7631.gbed76814.el9.x86_64 1/6 2023-12-02T02:45:52.094 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephsqlite-2:18.0.0-7631.gbed76814.el9.x86_64 2/6 2023-12-02T02:45:52.094 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-02T02:45:52.095 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-02T02:45:52.095 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-02T02:45:52.436 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7631.gbed76814.el9.noarc 7/7 2023-12-02T02:45:52.437 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7631.gbed76814.el9. 1/7 2023-12-02T02:45:52.437 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7631.gbed76814.el9.noarc 2/7 2023-12-02T02:45:52.437 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7631.gbed76814.el9.n 3/7 2023-12-02T02:45:52.437 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-02T02:45:52.437 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-02T02:45:52.437 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-02T02:45:52.514 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-02T02:45:52.514 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:45:52.515 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-02T02:45:52.515 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:52.515 INFO:teuthology.orchestra.run.smithi029.stdout: jq-1.6-16.el9.x86_64 2023-12-02T02:45:52.515 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:45:52.515 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-02T02:45:52.515 INFO:teuthology.orchestra.run.smithi029.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-02T02:45:52.515 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-02T02:45:52.515 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:45:52.515 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-02T02:45:52.757 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-02T02:45:52.757 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:52.757 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2023-12-02T02:45:52.757 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-grafana-dashboards-2:18.0.0-7631.gbed76814.el9.noarch 2023-12-02T02:45:52.757 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-dashboard-2:18.0.0-7631.gbed76814.el9.noarch 2023-12-02T02:45:52.757 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-prometheus-alerts-2:18.0.0-7631.gbed76814.el9.noarch 2023-12-02T02:45:52.757 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-02T02:45:52.757 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-02T02:45:52.758 INFO:teuthology.orchestra.run.smithi057.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-02T02:45:52.758 INFO:teuthology.orchestra.run.smithi057.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-02T02:45:52.758 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:52.758 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-12-02T02:45:52.861 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph 2023-12-02T02:45:52.976 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-02T02:45:53.464 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:05 ago on Sat 02 Dec 2023 02:44:48 AM UTC. 2023-12-02T02:45:53.582 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:12 ago on Sat 02 Dec 2023 02:44:41 AM UTC. 2023-12-02T02:45:53.623 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-02T02:45:53.625 INFO:teuthology.orchestra.run.smithi029.stdout:========================================================================================= 2023-12-02T02:45:53.625 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-12-02T02:45:53.625 INFO:teuthology.orchestra.run.smithi029.stdout:========================================================================================= 2023-12-02T02:45:53.626 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-02T02:45:53.626 INFO:teuthology.orchestra.run.smithi029.stdout: ceph x86_64 2:18.0.0-7631.gbed76814.el9 ceph 6.6 k 2023-12-02T02:45:53.626 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-12-02T02:45:53.626 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds x86_64 2:18.0.0-7631.gbed76814.el9 ceph 2.1 M 2023-12-02T02:45:53.626 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr x86_64 2:18.0.0-7631.gbed76814.el9 ceph 1.5 M 2023-12-02T02:45:53.626 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7631.gbed76814.el9 ceph-noarch 243 k 2023-12-02T02:45:53.626 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon x86_64 2:18.0.0-7631.gbed76814.el9 ceph 4.4 M 2023-12-02T02:45:53.626 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd x86_64 2:18.0.0-7631.gbed76814.el9 ceph 18 M 2023-12-02T02:45:53.627 INFO:teuthology.orchestra.run.smithi029.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-02T02:45:53.627 INFO:teuthology.orchestra.run.smithi029.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-12-02T02:45:53.627 INFO:teuthology.orchestra.run.smithi029.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-02T02:45:53.627 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-02T02:45:53.627 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-02T02:45:53.627 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-02T02:45:53.627 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-02T02:45:53.627 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-02T02:45:53.627 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-02T02:45:53.628 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-02T02:45:53.628 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-02T02:45:53.628 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-02T02:45:53.628 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-02T02:45:53.628 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-02T02:45:53.628 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-02T02:45:53.628 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-02T02:45:53.629 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-02T02:45:53.629 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-02T02:45:53.629 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-02T02:45:53.629 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-02T02:45:53.629 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-02T02:45:53.629 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-02T02:45:53.629 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-02T02:45:53.630 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-02T02:45:53.630 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-02T02:45:53.630 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-02T02:45:53.630 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-02T02:45:53.630 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-02T02:45:53.630 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-02T02:45:53.630 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-02T02:45:53.631 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:45:53.631 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-02T02:45:53.631 INFO:teuthology.orchestra.run.smithi029.stdout:========================================================================================= 2023-12-02T02:45:53.631 INFO:teuthology.orchestra.run.smithi029.stdout:Install 36 Packages 2023-12-02T02:45:53.631 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:45:53.632 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 30 M 2023-12-02T02:45:53.632 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 111 M 2023-12-02T02:45:53.632 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-02T02:45:53.736 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-12-02T02:45:53.737 INFO:teuthology.orchestra.run.smithi057.stdout:============================================================================================ 2023-12-02T02:45:53.737 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2023-12-02T02:45:53.737 INFO:teuthology.orchestra.run.smithi057.stdout:============================================================================================ 2023-12-02T02:45:53.737 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2023-12-02T02:45:53.737 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7631.gbed76814.el9 ceph-noarch 7.4 M 2023-12-02T02:45:53.738 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2023-12-02T02:45:53.738 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-02T02:45:53.738 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-02T02:45:53.738 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-02T02:45:53.738 INFO:teuthology.orchestra.run.smithi057.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-02T02:45:53.738 INFO:teuthology.orchestra.run.smithi057.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-02T02:45:53.738 INFO:teuthology.orchestra.run.smithi057.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-02T02:45:53.738 INFO:teuthology.orchestra.run.smithi057.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-02T02:45:53.738 INFO:teuthology.orchestra.run.smithi057.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-02T02:45:53.738 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-02T02:45:53.738 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-02T02:45:53.738 INFO:teuthology.orchestra.run.smithi057.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-02T02:45:53.738 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:53.738 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2023-12-02T02:45:53.739 INFO:teuthology.orchestra.run.smithi057.stdout:============================================================================================ 2023-12-02T02:45:53.739 INFO:teuthology.orchestra.run.smithi057.stdout:Install 12 Packages 2023-12-02T02:45:53.739 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:45:53.739 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 37 M 2023-12-02T02:45:53.739 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 209 M 2023-12-02T02:45:53.739 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2023-12-02T02:45:54.152 INFO:teuthology.orchestra.run.smithi029.stdout:(1/36): ceph-18.0.0-7631.gbed76814.el9.x86_64.r 31 kB/s | 6.6 kB 00:00 2023-12-02T02:45:54.162 INFO:teuthology.orchestra.run.smithi057.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 234 kB/s | 32 kB 00:00 2023-12-02T02:45:54.204 INFO:teuthology.orchestra.run.smithi057.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 427 kB/s | 17 kB 00:00 2023-12-02T02:45:54.286 INFO:teuthology.orchestra.run.smithi029.stdout:(2/36): ceph-mgr-18.0.0-7631.gbed76814.el9.x86_ 4.2 MB/s | 1.5 MB 00:00 2023-12-02T02:45:54.296 INFO:teuthology.orchestra.run.smithi057.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2023-12-02T02:45:54.313 INFO:teuthology.orchestra.run.smithi057.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 319 kB/s | 34 kB 00:00 2023-12-02T02:45:54.589 INFO:teuthology.orchestra.run.smithi057.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 16 MB/s | 4.7 MB 00:00 2023-12-02T02:45:54.647 INFO:teuthology.orchestra.run.smithi057.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 4.8 MB 00:00 2023-12-02T02:45:54.672 INFO:teuthology.orchestra.run.smithi057.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2023-12-02T02:45:54.931 INFO:teuthology.orchestra.run.smithi057.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 3.0 MB/s | 803 kB 00:00 2023-12-02T02:45:55.248 INFO:teuthology.orchestra.run.smithi057.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2023-12-02T02:45:55.261 INFO:teuthology.orchestra.run.smithi029.stdout:(3/36): ceph-osd-18.0.0-7631.gbed76814.el9.x86_ 18 MB/s | 18 MB 00:00 2023-12-02T02:45:55.273 INFO:teuthology.orchestra.run.smithi057.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 575 kB/s | 195 kB 00:00 2023-12-02T02:45:55.312 INFO:teuthology.orchestra.run.smithi029.stdout:(4/36): ceph-mgr-modules-core-18.0.0-7631.gbed7 4.8 MB/s | 243 kB 00:00 2023-12-02T02:45:55.429 INFO:teuthology.orchestra.run.smithi029.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2023-12-02T02:45:55.488 INFO:teuthology.orchestra.run.smithi029.stdout:(6/36): ceph-mds-18.0.0-7631.gbed76814.el9.x86_ 1.3 MB/s | 2.1 MB 00:01 2023-12-02T02:45:55.490 INFO:teuthology.orchestra.run.smithi057.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2023-12-02T02:45:55.523 INFO:teuthology.orchestra.run.smithi029.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.5 MB/s | 144 kB 00:00 2023-12-02T02:45:55.549 INFO:teuthology.orchestra.run.smithi029.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 523 kB/s | 32 kB 00:00 2023-12-02T02:45:55.583 INFO:teuthology.orchestra.run.smithi029.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 800 kB/s | 47 kB 00:00 2023-12-02T02:45:55.616 INFO:teuthology.orchestra.run.smithi029.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 575 kB/s | 39 kB 00:00 2023-12-02T02:45:55.650 INFO:teuthology.orchestra.run.smithi029.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2023-12-02T02:45:55.726 INFO:teuthology.orchestra.run.smithi029.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2023-12-02T02:45:55.752 INFO:teuthology.orchestra.run.smithi029.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 103 kB 00:00 2023-12-02T02:45:55.777 INFO:teuthology.orchestra.run.smithi029.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2023-12-02T02:45:55.811 INFO:teuthology.orchestra.run.smithi029.stdout:(15/36): python3-requests-2.25.1-7.el9.noarch.r 1.9 MB/s | 113 kB 00:00 2023-12-02T02:45:55.836 INFO:teuthology.orchestra.run.smithi029.stdout:(16/36): python3-urllib3-1.26.5-4.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2023-12-02T02:45:56.148 INFO:teuthology.orchestra.run.smithi057.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-7 3.5 MB/s | 7.4 MB 00:02 2023-12-02T02:45:56.151 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:45:56.151 INFO:teuthology.orchestra.run.smithi057.stdout:Total 15 MB/s | 37 MB 00:02 2023-12-02T02:45:56.165 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2023-12-02T02:45:56.191 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2023-12-02T02:45:56.191 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2023-12-02T02:45:56.245 INFO:teuthology.orchestra.run.smithi029.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 370 kB/s | 151 kB 00:00 2023-12-02T02:45:56.354 INFO:teuthology.orchestra.run.smithi029.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 400 kB/s | 43 kB 00:00 2023-12-02T02:45:56.513 INFO:teuthology.orchestra.run.smithi029.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2023-12-02T02:45:56.539 INFO:teuthology.orchestra.run.smithi029.stdout:(20/36): lua-devel-5.4.4-4.el9.x86_64.rpm 31 kB/s | 22 kB 00:00 2023-12-02T02:45:56.591 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2023-12-02T02:45:56.591 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2023-12-02T02:45:56.714 INFO:teuthology.orchestra.run.smithi029.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2023-12-02T02:45:56.816 INFO:teuthology.orchestra.run.smithi029.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 174 kB/s | 18 kB 00:00 2023-12-02T02:45:56.842 INFO:teuthology.orchestra.run.smithi029.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 35 kB/s | 11 kB 00:00 2023-12-02T02:45:56.918 INFO:teuthology.orchestra.run.smithi029.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 230 kB/s | 23 kB 00:00 2023-12-02T02:45:56.968 INFO:teuthology.orchestra.run.smithi029.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 154 kB/s | 19 kB 00:00 2023-12-02T02:45:57.010 INFO:teuthology.orchestra.run.smithi029.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 213 kB/s | 20 kB 00:00 2023-12-02T02:45:57.111 INFO:teuthology.orchestra.run.smithi029.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 787 kB/s | 79 kB 00:00 2023-12-02T02:45:57.161 INFO:teuthology.orchestra.run.smithi029.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 240 kB/s | 46 kB 00:00 2023-12-02T02:45:57.245 INFO:teuthology.orchestra.run.smithi029.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2023-12-02T02:45:57.271 INFO:teuthology.orchestra.run.smithi029.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 150 kB/s | 16 kB 00:00 2023-12-02T02:45:57.335 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2023-12-02T02:45:57.346 INFO:teuthology.orchestra.run.smithi029.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 893 kB/s | 90 kB 00:00 2023-12-02T02:45:57.380 INFO:teuthology.orchestra.run.smithi029.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 328 kB/s | 36 kB 00:00 2023-12-02T02:45:57.405 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-02T02:45:57.445 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-02T02:45:57.449 INFO:teuthology.orchestra.run.smithi029.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2023-12-02T02:45:57.523 INFO:teuthology.orchestra.run.smithi029.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 266 kB/s | 20 kB 00:00 2023-12-02T02:45:57.545 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-02T02:45:57.589 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-02T02:45:57.623 INFO:teuthology.orchestra.run.smithi029.stdout:(35/36): ceph-mon-18.0.0-7631.gbed76814.el9.x86 1.3 MB/s | 4.4 MB 00:03 2023-12-02T02:45:57.657 INFO:teuthology.orchestra.run.smithi029.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.5 MB/s | 427 kB 00:00 2023-12-02T02:45:57.663 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:45:57.663 INFO:teuthology.orchestra.run.smithi029.stdout:Total 7.5 MB/s | 30 MB 00:04 2023-12-02T02:45:57.677 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-02T02:45:57.725 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-02T02:45:57.725 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-02T02:45:57.776 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-02T02:45:57.805 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-02T02:45:57.896 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-02T02:45:58.194 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-02T02:45:58.194 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-02T02:45:58.581 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-02T02:45:58.705 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-02T02:45:59.555 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-02T02:45:59.621 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2023-12-02T02:45:59.663 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2023-12-02T02:45:59.700 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2023-12-02T02:45:59.784 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2023-12-02T02:45:59.834 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2023-12-02T02:45:59.875 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2023-12-02T02:45:59.975 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2023-12-02T02:46:00.060 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2023-12-02T02:46:00.107 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2023-12-02T02:46:00.113 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-02T02:46:00.261 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2023-12-02T02:46:00.318 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 11/36 2023-12-02T02:46:00.365 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2023-12-02T02:46:00.442 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2023-12-02T02:46:00.518 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2023-12-02T02:46:00.728 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2023-12-02T02:46:00.790 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2023-12-02T02:46:00.838 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2023-12-02T02:46:00.883 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2023-12-02T02:46:00.921 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2023-12-02T02:46:01.003 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2023-12-02T02:46:01.041 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-02T02:46:01.084 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2023-12-02T02:46:01.125 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2023-12-02T02:46:01.171 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2023-12-02T02:46:01.206 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2023-12-02T02:46:01.294 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-02T02:46:01.300 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2023-12-02T02:46:01.320 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7631.gbed76 12/12 2023-12-02T02:46:01.353 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2023-12-02T02:46:01.431 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2023-12-02T02:46:01.582 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2023-12-02T02:46:01.716 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-02T02:46:01.879 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7631.gbed76814.el9. 31/36 2023-12-02T02:46:01.907 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-2:18.0.0-7631.gbed76814.el9.x86_64 32/36 2023-12-02T02:46:01.934 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7631.gbed76814.el9.x86_64 32/36 2023-12-02T02:46:01.934 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T02:46:01.934 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-02T02:46:01.934 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-02T02:46:01.934 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-02T02:46:01.934 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:03.552 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-osd-2:18.0.0-7631.gbed76814.el9.x86_64 33/36 2023-12-02T02:46:03.580 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-osd-2:18.0.0-7631.gbed76814.el9.x86_64 33/36 2023-12-02T02:46:03.580 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T02:46:03.580 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-02T02:46:03.580 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-02T02:46:03.580 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-02T02:46:03.580 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:03.922 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7631.gbed76 12/12 2023-12-02T02:46:03.923 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7631.gbed76 1/12 2023-12-02T02:46:03.923 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-02T02:46:03.923 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-02T02:46:03.923 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-02T02:46:03.923 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-02T02:46:03.923 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-02T02:46:03.923 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-02T02:46:03.923 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-02T02:46:03.923 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-02T02:46:03.923 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-02T02:46:03.923 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-02T02:46:04.053 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mon-2:18.0.0-7631.gbed76814.el9.x86_64 34/36 2023-12-02T02:46:04.078 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mon-2:18.0.0-7631.gbed76814.el9.x86_64 34/36 2023-12-02T02:46:04.078 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T02:46:04.078 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-02T02:46:04.078 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-02T02:46:04.078 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-02T02:46:04.078 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:04.288 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mds-2:18.0.0-7631.gbed76814.el9.x86_64 35/36 2023-12-02T02:46:04.312 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mds-2:18.0.0-7631.gbed76814.el9.x86_64 35/36 2023-12-02T02:46:04.312 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T02:46:04.312 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-02T02:46:04.312 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-02T02:46:04.313 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-02T02:46:04.313 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:04.380 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-2:18.0.0-7631.gbed76814.el9.x86_64 36/36 2023-12-02T02:46:04.516 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-02T02:46:04.516 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:04.516 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2023-12-02T02:46:04.516 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7631.gbed76814.el9.noarch 2023-12-02T02:46:04.516 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-02T02:46:04.516 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-02T02:46:04.516 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-02T02:46:04.516 INFO:teuthology.orchestra.run.smithi057.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-02T02:46:04.516 INFO:teuthology.orchestra.run.smithi057.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-02T02:46:04.516 INFO:teuthology.orchestra.run.smithi057.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-02T02:46:04.516 INFO:teuthology.orchestra.run.smithi057.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-02T02:46:04.516 INFO:teuthology.orchestra.run.smithi057.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-02T02:46:04.517 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-02T02:46:04.517 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-02T02:46:04.517 INFO:teuthology.orchestra.run.smithi057.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-02T02:46:04.517 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:04.517 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-12-02T02:46:04.892 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-mgr-rook 2023-12-02T02:46:05.499 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:24 ago on Sat 02 Dec 2023 02:44:41 AM UTC. 2023-12-02T02:46:05.650 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-12-02T02:46:05.652 INFO:teuthology.orchestra.run.smithi057.stdout:======================================================================================== 2023-12-02T02:46:05.652 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2023-12-02T02:46:05.652 INFO:teuthology.orchestra.run.smithi057.stdout:======================================================================================== 2023-12-02T02:46:05.652 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2023-12-02T02:46:05.652 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-rook noarch 2:18.0.0-7631.gbed76814.el9 ceph-noarch 50 k 2023-12-02T02:46:05.652 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2023-12-02T02:46:05.652 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-02T02:46:05.652 INFO:teuthology.orchestra.run.smithi057.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-02T02:46:05.652 INFO:teuthology.orchestra.run.smithi057.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-02T02:46:05.652 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-02T02:46:05.652 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-02T02:46:05.652 INFO:teuthology.orchestra.run.smithi057.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-02T02:46:05.653 INFO:teuthology.orchestra.run.smithi057.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-02T02:46:05.653 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-02T02:46:05.653 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-02T02:46:05.653 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-02T02:46:05.653 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-02T02:46:05.653 INFO:teuthology.orchestra.run.smithi057.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-02T02:46:05.653 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:05.653 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2023-12-02T02:46:05.653 INFO:teuthology.orchestra.run.smithi057.stdout:======================================================================================== 2023-12-02T02:46:05.653 INFO:teuthology.orchestra.run.smithi057.stdout:Install 13 Packages 2023-12-02T02:46:05.653 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:05.654 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 2.0 M 2023-12-02T02:46:05.654 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 26 M 2023-12-02T02:46:05.654 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2023-12-02T02:46:06.160 INFO:teuthology.orchestra.run.smithi057.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 114 kB/s | 18 kB 00:00 2023-12-02T02:46:06.186 INFO:teuthology.orchestra.run.smithi057.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 133 kB/s | 24 kB 00:00 2023-12-02T02:46:06.394 INFO:teuthology.orchestra.run.smithi057.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 750 kB/s | 174 kB 00:00 2023-12-02T02:46:06.419 INFO:teuthology.orchestra.run.smithi057.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 566 kB/s | 132 kB 00:00 2023-12-02T02:46:06.445 INFO:teuthology.orchestra.run.smithi057.stdout:(5/13): ceph-mgr-rook-18.0.0-7631.gbed76814.el9 113 kB/s | 50 kB 00:00 2023-12-02T02:46:06.470 INFO:teuthology.orchestra.run.smithi057.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2023-12-02T02:46:06.495 INFO:teuthology.orchestra.run.smithi057.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 566 kB/s | 43 kB 00:00 2023-12-02T02:46:06.587 INFO:teuthology.orchestra.run.smithi057.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 128 kB/s | 15 kB 00:00 2023-12-02T02:46:06.612 INFO:teuthology.orchestra.run.smithi057.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 192 kB/s | 32 kB 00:00 2023-12-02T02:46:06.720 INFO:teuthology.orchestra.run.smithi057.stdout:(10/13): python3-google-auth-2.23.4-1.el9.noarc 876 kB/s | 197 kB 00:00 2023-12-02T02:46:06.746 INFO:teuthology.orchestra.run.smithi057.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 442 kB/s | 59 kB 00:00 2023-12-02T02:46:06.813 INFO:teuthology.orchestra.run.smithi057.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 976 kB/s | 90 kB 00:00 2023-12-02T02:46:06.871 INFO:teuthology.orchestra.run.smithi057.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.6 MB/s | 1.0 MB 00:00 2023-12-02T02:46:06.873 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:46:06.873 INFO:teuthology.orchestra.run.smithi057.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2023-12-02T02:46:06.874 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2023-12-02T02:46:06.889 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2023-12-02T02:46:06.889 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2023-12-02T02:46:06.919 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-2:18.0.0-7631.gbed76814.el9.x86_64 36/36 2023-12-02T02:46:06.919 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-2:18.0.0-7631.gbed76814.el9.x86_64 1/36 2023-12-02T02:46:06.919 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mds-2:18.0.0-7631.gbed76814.el9.x86_64 2/36 2023-12-02T02:46:06.919 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-2:18.0.0-7631.gbed76814.el9.x86_64 3/36 2023-12-02T02:46:06.919 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mon-2:18.0.0-7631.gbed76814.el9.x86_64 4/36 2023-12-02T02:46:06.919 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-osd-2:18.0.0-7631.gbed76814.el9.x86_64 5/36 2023-12-02T02:46:06.919 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7631.gbed76814.el9. 6/36 2023-12-02T02:46:06.920 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2023-12-02T02:46:06.920 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2023-12-02T02:46:06.920 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2023-12-02T02:46:06.920 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2023-12-02T02:46:06.920 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2023-12-02T02:46:06.920 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2023-12-02T02:46:06.920 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2023-12-02T02:46:06.920 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2023-12-02T02:46:06.920 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2023-12-02T02:46:06.920 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2023-12-02T02:46:06.920 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/36 2023-12-02T02:46:06.920 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2023-12-02T02:46:06.920 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2023-12-02T02:46:06.920 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2023-12-02T02:46:06.921 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-02T02:46:06.921 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2023-12-02T02:46:06.921 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2023-12-02T02:46:06.921 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2023-12-02T02:46:06.921 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2023-12-02T02:46:06.922 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2023-12-02T02:46:06.922 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2023-12-02T02:46:06.922 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2023-12-02T02:46:06.922 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2023-12-02T02:46:06.922 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-02T02:46:06.922 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2023-12-02T02:46:06.923 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2023-12-02T02:46:06.923 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2023-12-02T02:46:06.923 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2023-12-02T02:46:06.923 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2023-12-02T02:46:07.000 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2023-12-02T02:46:07.001 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2023-12-02T02:46:07.549 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2023-12-02T02:46:07.728 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-02T02:46:07.792 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-02T02:46:07.856 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-02T02:46:07.892 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-02T02:46:07.934 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-02T02:46:07.948 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2023-12-02T02:46:07.948 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:07.948 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-02T02:46:07.948 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:46:07.949 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:46:07.949 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:46:07.949 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core-2:18.0.0-7631.gbed76814.el9.noarch 2023-12-02T02:46:07.949 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:46:07.949 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:46:07.949 INFO:teuthology.orchestra.run.smithi029.stdout: lua-5.4.4-4.el9.x86_64 2023-12-02T02:46:07.949 INFO:teuthology.orchestra.run.smithi029.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-12-02T02:46:07.949 INFO:teuthology.orchestra.run.smithi029.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-02T02:46:07.949 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-02T02:46:07.949 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-02T02:46:07.949 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-02T02:46:07.949 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-02T02:46:07.949 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-02T02:46:07.950 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-02T02:46:07.950 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-02T02:46:07.950 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-02T02:46:07.950 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-02T02:46:07.950 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-02T02:46:07.950 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-02T02:46:07.950 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-02T02:46:07.950 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-02T02:46:07.950 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-02T02:46:07.950 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-02T02:46:07.950 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply-3.11-14.el9.noarch 2023-12-02T02:46:07.950 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-02T02:46:07.950 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-02T02:46:07.951 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-02T02:46:07.951 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-02T02:46:07.951 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-02T02:46:07.951 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-02T02:46:07.951 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-02T02:46:07.951 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-02T02:46:07.951 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-02T02:46:07.951 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-02T02:46:07.951 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-02T02:46:07.951 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:07.951 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-02T02:46:08.030 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-02T02:46:08.134 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-02T02:46:08.186 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-02T02:46:08.446 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-base 2023-12-02T02:46:08.945 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-02T02:46:08.998 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-02T02:46:09.032 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-02T02:46:09.047 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:21 ago on Sat 02 Dec 2023 02:44:48 AM UTC. 2023-12-02T02:46:09.080 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-02T02:46:09.101 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-rook-2:18.0.0-7631.gbed76814.el9.noarch 13/13 2023-12-02T02:46:09.147 INFO:teuthology.orchestra.run.smithi029.stdout:Package ceph-base-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T02:46:09.199 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-02T02:46:09.200 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-12-02T02:46:09.200 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-02T02:46:09.274 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install cephadm 2023-12-02T02:46:09.867 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7631.gbed76814.el9.noarch 13/13 2023-12-02T02:46:09.867 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7631.gbed76814.el9.noarch 1/13 2023-12-02T02:46:09.867 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-02T02:46:09.867 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-02T02:46:09.867 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-02T02:46:09.867 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-02T02:46:09.867 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-02T02:46:09.867 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-02T02:46:09.867 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-02T02:46:09.868 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-02T02:46:09.868 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-02T02:46:09.868 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-02T02:46:09.868 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-02T02:46:09.868 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:21 ago on Sat 02 Dec 2023 02:44:48 AM UTC. 2023-12-02T02:46:10.019 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-02T02:46:10.020 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:46:10.020 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-12-02T02:46:10.020 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:46:10.020 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-02T02:46:10.020 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm noarch 2:18.0.0-7631.gbed76814.el9 ceph-noarch 776 k 2023-12-02T02:46:10.020 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:10.020 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-02T02:46:10.020 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:46:10.020 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-12-02T02:46:10.020 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:10.020 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 776 k 2023-12-02T02:46:10.020 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 790 k 2023-12-02T02:46:10.020 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-02T02:46:10.289 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-02T02:46:10.289 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:10.290 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2023-12-02T02:46:10.290 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-rook-2:18.0.0-7631.gbed76814.el9.noarch 2023-12-02T02:46:10.290 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-02T02:46:10.290 INFO:teuthology.orchestra.run.smithi057.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-02T02:46:10.290 INFO:teuthology.orchestra.run.smithi057.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-02T02:46:10.290 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-02T02:46:10.290 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-02T02:46:10.290 INFO:teuthology.orchestra.run.smithi057.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-02T02:46:10.290 INFO:teuthology.orchestra.run.smithi057.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-02T02:46:10.290 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-02T02:46:10.290 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-02T02:46:10.290 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-02T02:46:10.290 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-02T02:46:10.291 INFO:teuthology.orchestra.run.smithi057.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-02T02:46:10.291 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:10.291 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-12-02T02:46:10.375 INFO:teuthology.orchestra.run.smithi029.stdout:cephadm-18.0.0-7631.gbed76814.el9.noarch.rpm 2.1 MB/s | 776 kB 00:00 2023-12-02T02:46:10.376 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:46:10.376 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.1 MB/s | 776 kB 00:00 2023-12-02T02:46:10.376 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-02T02:46:10.382 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-02T02:46:10.382 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-02T02:46:10.388 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-02T02:46:10.388 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-02T02:46:10.465 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-02T02:46:10.564 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-mgr-cephadm 2023-12-02T02:46:10.955 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:18.0.0-7631.gbed76814.el9.noarch 1/1 2023-12-02T02:46:11.047 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephadm-2:18.0.0-7631.gbed76814.el9.noarch 1/1 2023-12-02T02:46:11.170 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:30 ago on Sat 02 Dec 2023 02:44:41 AM UTC. 2023-12-02T02:46:11.323 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-12-02T02:46:11.324 INFO:teuthology.orchestra.run.smithi057.stdout:======================================================================================== 2023-12-02T02:46:11.324 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2023-12-02T02:46:11.324 INFO:teuthology.orchestra.run.smithi057.stdout:======================================================================================== 2023-12-02T02:46:11.324 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2023-12-02T02:46:11.324 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7631.gbed76814.el9 ceph-noarch 133 k 2023-12-02T02:46:11.324 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2023-12-02T02:46:11.324 INFO:teuthology.orchestra.run.smithi057.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-02T02:46:11.324 INFO:teuthology.orchestra.run.smithi057.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-02T02:46:11.324 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-02T02:46:11.324 INFO:teuthology.orchestra.run.smithi057.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-02T02:46:11.324 INFO:teuthology.orchestra.run.smithi057.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-02T02:46:11.324 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:11.325 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2023-12-02T02:46:11.325 INFO:teuthology.orchestra.run.smithi057.stdout:======================================================================================== 2023-12-02T02:46:11.325 INFO:teuthology.orchestra.run.smithi057.stdout:Install 6 Packages 2023-12-02T02:46:11.325 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:11.325 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 6.8 M 2023-12-02T02:46:11.325 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 33 M 2023-12-02T02:46:11.325 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2023-12-02T02:46:11.532 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:18.0.0-7631.gbed76814.el9.noarch 1/1 2023-12-02T02:46:11.836 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephadm-2:18.0.0-7631.gbed76814.el9.noarch 1/1 2023-12-02T02:46:11.836 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:11.836 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-02T02:46:11.836 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm-2:18.0.0-7631.gbed76814.el9.noarch 2023-12-02T02:46:11.836 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:11.836 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-02T02:46:11.850 INFO:teuthology.orchestra.run.smithi057.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2023-12-02T02:46:12.033 INFO:teuthology.orchestra.run.smithi057.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2023-12-02T02:46:12.109 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-immutable-object-cache 2023-12-02T02:46:12.125 INFO:teuthology.orchestra.run.smithi057.stdout:(3/6): ceph-mgr-cephadm-18.0.0-7631.gbed76814.e 297 kB/s | 133 kB 00:00 2023-12-02T02:46:12.192 INFO:teuthology.orchestra.run.smithi057.stdout:(4/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2023-12-02T02:46:12.209 INFO:teuthology.orchestra.run.smithi057.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 329 kB/s | 58 kB 00:00 2023-12-02T02:46:12.276 INFO:teuthology.orchestra.run.smithi057.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 340 kB/s | 51 kB 00:00 2023-12-02T02:46:12.280 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:46:12.280 INFO:teuthology.orchestra.run.smithi057.stdout:Total 7.1 MB/s | 6.8 MB 00:00 2023-12-02T02:46:12.280 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2023-12-02T02:46:12.290 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2023-12-02T02:46:12.290 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2023-12-02T02:46:12.386 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2023-12-02T02:46:12.387 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2023-12-02T02:46:12.694 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2023-12-02T02:46:12.723 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:24 ago on Sat 02 Dec 2023 02:44:48 AM UTC. 2023-12-02T02:46:12.865 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-02T02:46:12.875 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-02T02:46:12.876 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:46:12.876 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-12-02T02:46:12.876 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:46:12.876 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-02T02:46:12.876 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7631.gbed76814.el9 ceph 141 k 2023-12-02T02:46:12.876 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:12.876 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-02T02:46:12.876 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:46:12.876 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-12-02T02:46:12.876 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:12.877 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 141 k 2023-12-02T02:46:12.877 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 400 k 2023-12-02T02:46:12.877 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-02T02:46:12.912 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-02T02:46:13.099 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-immutable-object-cache-18.0.0-7631.gbed768 636 kB/s | 141 kB 00:00 2023-12-02T02:46:13.100 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:46:13.100 INFO:teuthology.orchestra.run.smithi029.stdout:Total 633 kB/s | 141 kB 00:00 2023-12-02T02:46:13.100 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-02T02:46:13.108 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-02T02:46:13.108 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-02T02:46:13.163 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-02T02:46:13.163 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-02T02:46:13.306 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-02T02:46:13.384 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7631.gbed76814. 1/1 2023-12-02T02:46:13.410 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7631.gbed76814. 1/1 2023-12-02T02:46:13.410 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T02:46:13.410 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-02T02:46:13.410 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:13.464 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-02T02:46:13.541 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-02T02:46:13.609 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-02T02:46:13.640 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7631.gbed76814.el9.noarch 6/6 2023-12-02T02:46:14.399 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7631.gbed76814.el9.noarch 6/6 2023-12-02T02:46:14.399 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7631.gbed76814.el9.noarch 1/6 2023-12-02T02:46:14.399 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-02T02:46:14.399 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-02T02:46:14.400 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-02T02:46:14.400 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-02T02:46:14.487 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7631.gbed76814. 1/1 2023-12-02T02:46:14.487 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:14.487 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-02T02:46:14.487 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:46:14.487 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:14.488 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-02T02:46:14.757 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-02T02:46:14.757 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:14.757 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2023-12-02T02:46:14.757 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-cephadm-2:18.0.0-7631.gbed76814.el9.noarch 2023-12-02T02:46:14.757 INFO:teuthology.orchestra.run.smithi057.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-02T02:46:14.757 INFO:teuthology.orchestra.run.smithi057.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-02T02:46:14.757 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-02T02:46:14.758 INFO:teuthology.orchestra.run.smithi057.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-02T02:46:14.758 INFO:teuthology.orchestra.run.smithi057.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-02T02:46:14.758 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:14.758 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-12-02T02:46:14.801 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr 2023-12-02T02:46:14.993 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-fuse 2023-12-02T02:46:15.407 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:27 ago on Sat 02 Dec 2023 02:44:48 AM UTC. 2023-12-02T02:46:15.506 INFO:teuthology.orchestra.run.smithi029.stdout:Package ceph-mgr-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T02:46:15.559 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-02T02:46:15.560 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-12-02T02:46:15.560 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-02T02:46:15.599 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:34 ago on Sat 02 Dec 2023 02:44:41 AM UTC. 2023-12-02T02:46:15.635 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-dashboard 2023-12-02T02:46:15.753 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-12-02T02:46:15.754 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:46:15.754 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2023-12-02T02:46:15.754 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:46:15.754 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2023-12-02T02:46:15.754 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-fuse x86_64 2:18.0.0-7631.gbed76814.el9 ceph 859 k 2023-12-02T02:46:15.754 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2023-12-02T02:46:15.754 INFO:teuthology.orchestra.run.smithi057.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-02T02:46:15.754 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:15.754 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2023-12-02T02:46:15.754 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:46:15.755 INFO:teuthology.orchestra.run.smithi057.stdout:Install 2 Packages 2023-12-02T02:46:15.755 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:15.755 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 938 k 2023-12-02T02:46:15.755 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 2.7 M 2023-12-02T02:46:15.755 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2023-12-02T02:46:15.886 INFO:teuthology.orchestra.run.smithi057.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 605 kB/s | 79 kB 00:00 2023-12-02T02:46:16.235 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:28 ago on Sat 02 Dec 2023 02:44:48 AM UTC. 2023-12-02T02:46:16.310 INFO:teuthology.orchestra.run.smithi057.stdout:(2/2): ceph-fuse-18.0.0-7631.gbed76814.el9.x86_ 1.5 MB/s | 859 kB 00:00 2023-12-02T02:46:16.312 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:46:16.312 INFO:teuthology.orchestra.run.smithi057.stdout:Total 1.6 MB/s | 938 kB 00:00 2023-12-02T02:46:16.312 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2023-12-02T02:46:16.334 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2023-12-02T02:46:16.335 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2023-12-02T02:46:16.386 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-02T02:46:16.387 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:46:16.387 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-12-02T02:46:16.387 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:46:16.387 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-02T02:46:16.387 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7631.gbed76814.el9 ceph-noarch 1.8 M 2023-12-02T02:46:16.387 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-12-02T02:46:16.387 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7631.gbed76814.el9 ceph-noarch 28 k 2023-12-02T02:46:16.387 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7631.gbed76814.el9 ceph-noarch 15 k 2023-12-02T02:46:16.387 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-02T02:46:16.387 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-02T02:46:16.387 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-02T02:46:16.388 INFO:teuthology.orchestra.run.smithi029.stdout:Installing weak dependencies: 2023-12-02T02:46:16.388 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-02T02:46:16.388 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:16.388 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-02T02:46:16.388 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:46:16.388 INFO:teuthology.orchestra.run.smithi029.stdout:Install 7 Packages 2023-12-02T02:46:16.388 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:16.388 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 2.1 M 2023-12-02T02:46:16.389 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 8.7 M 2023-12-02T02:46:16.389 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-02T02:46:16.414 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2023-12-02T02:46:16.415 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2023-12-02T02:46:16.605 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2023-12-02T02:46:16.635 INFO:teuthology.orchestra.run.smithi029.stdout:(1/7): ceph-grafana-dashboards-18.0.0-7631.gbed 143 kB/s | 28 kB 00:00 2023-12-02T02:46:16.736 INFO:teuthology.orchestra.run.smithi029.stdout:(2/7): ceph-prometheus-alerts-18.0.0-7631.gbed7 51 kB/s | 15 kB 00:00 2023-12-02T02:46:16.748 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-02T02:46:16.830 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-fuse-2:18.0.0-7631.gbed76814.el9.x86_64 2/2 2023-12-02T02:46:16.978 INFO:teuthology.orchestra.run.smithi029.stdout:(3/7): ceph-mgr-dashboard-18.0.0-7631.gbed76814 3.3 MB/s | 1.8 MB 00:00 2023-12-02T02:46:17.004 INFO:teuthology.orchestra.run.smithi029.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 25 kB/s | 9.0 kB 00:00 2023-12-02T02:46:17.221 INFO:teuthology.orchestra.run.smithi029.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 89 kB/s | 41 kB 00:00 2023-12-02T02:46:17.413 INFO:teuthology.orchestra.run.smithi029.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 71 kB/s | 31 kB 00:00 2023-12-02T02:46:17.438 INFO:teuthology.orchestra.run.smithi029.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 434 kB/s | 188 kB 00:00 2023-12-02T02:46:17.443 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:46:17.444 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.0 MB/s | 2.1 MB 00:01 2023-12-02T02:46:17.444 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-02T02:46:17.452 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-02T02:46:17.452 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-02T02:46:17.525 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-02T02:46:17.525 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-02T02:46:17.546 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-fuse-2:18.0.0-7631.gbed76814.el9.x86_64 2/2 2023-12-02T02:46:17.546 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-fuse-2:18.0.0-7631.gbed76814.el9.x86_64 1/2 2023-12-02T02:46:17.832 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-02T02:46:17.842 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-02T02:46:17.843 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:17.843 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2023-12-02T02:46:17.843 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-fuse-2:18.0.0-7631.gbed76814.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-02T02:46:17.843 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:17.843 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-12-02T02:46:17.915 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-02T02:46:17.961 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-02T02:46:18.024 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-02T02:46:18.064 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-02T02:46:18.095 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7631.gbed76814.el9.n 5/7 2023-12-02T02:46:18.118 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-volume 2023-12-02T02:46:18.346 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7631.gbed76814.el9. 6/7 2023-12-02T02:46:18.389 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7631.gbed76814.el9.noarc 7/7 2023-12-02T02:46:18.724 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:37 ago on Sat 02 Dec 2023 02:44:41 AM UTC. 2023-12-02T02:46:18.876 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-12-02T02:46:18.877 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:46:18.877 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2023-12-02T02:46:18.877 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:46:18.877 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2023-12-02T02:46:18.877 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-volume noarch 2:18.0.0-7631.gbed76814.el9 ceph-noarch 262 k 2023-12-02T02:46:18.877 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:18.877 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2023-12-02T02:46:18.878 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:46:18.878 INFO:teuthology.orchestra.run.smithi057.stdout:Install 1 Package 2023-12-02T02:46:18.878 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:18.878 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 262 k 2023-12-02T02:46:18.878 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 1.2 M 2023-12-02T02:46:18.878 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2023-12-02T02:46:19.079 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7631.gbed76814.el9.noarc 7/7 2023-12-02T02:46:19.080 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7631.gbed76814.el9. 1/7 2023-12-02T02:46:19.080 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7631.gbed76814.el9.noarc 2/7 2023-12-02T02:46:19.080 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7631.gbed76814.el9.n 3/7 2023-12-02T02:46:19.080 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-02T02:46:19.080 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-02T02:46:19.080 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-02T02:46:19.348 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-volume-18.0.0-7631.gbed76814.el9.noarch.rp 559 kB/s | 262 kB 00:00 2023-12-02T02:46:19.349 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:46:19.349 INFO:teuthology.orchestra.run.smithi057.stdout:Total 557 kB/s | 262 kB 00:00 2023-12-02T02:46:19.349 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2023-12-02T02:46:19.358 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2023-12-02T02:46:19.358 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2023-12-02T02:46:19.393 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2023-12-02T02:46:19.394 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2023-12-02T02:46:19.421 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-02T02:46:19.421 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:19.421 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-02T02:46:19.421 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards-2:18.0.0-7631.gbed76814.el9.noarch 2023-12-02T02:46:19.421 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard-2:18.0.0-7631.gbed76814.el9.noarch 2023-12-02T02:46:19.421 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts-2:18.0.0-7631.gbed76814.el9.noarch 2023-12-02T02:46:19.421 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-02T02:46:19.421 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-02T02:46:19.422 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-02T02:46:19.422 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-02T02:46:19.422 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:19.422 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-02T02:46:19.624 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2023-12-02T02:46:19.683 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-volume-2:18.0.0-7631.gbed76814.el9.noarch 1/1 2023-12-02T02:46:19.699 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-volume-2:18.0.0-7631.gbed76814.el9.noarch 1/1 2023-12-02T02:46:19.700 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T02:46:19.700 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-02T02:46:19.700 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:19.719 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-02T02:46:20.326 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:32 ago on Sat 02 Dec 2023 02:44:48 AM UTC. 2023-12-02T02:46:20.486 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-02T02:46:20.487 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2023-12-02T02:46:20.487 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-12-02T02:46:20.487 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2023-12-02T02:46:20.487 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-02T02:46:20.487 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7631.gbed76814.el9 ceph-noarch 7.4 M 2023-12-02T02:46:20.487 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-12-02T02:46:20.487 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-02T02:46:20.487 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-02T02:46:20.487 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-02T02:46:20.487 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-02T02:46:20.487 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-02T02:46:20.487 INFO:teuthology.orchestra.run.smithi029.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-02T02:46:20.487 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-02T02:46:20.488 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-02T02:46:20.488 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-02T02:46:20.488 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-02T02:46:20.488 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-02T02:46:20.488 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:20.488 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-02T02:46:20.488 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2023-12-02T02:46:20.488 INFO:teuthology.orchestra.run.smithi029.stdout:Install 12 Packages 2023-12-02T02:46:20.488 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:20.489 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 37 M 2023-12-02T02:46:20.489 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 209 M 2023-12-02T02:46:20.489 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-02T02:46:20.790 INFO:teuthology.orchestra.run.smithi029.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 178 kB/s | 32 kB 00:00 2023-12-02T02:46:20.824 INFO:teuthology.orchestra.run.smithi029.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 514 kB/s | 17 kB 00:00 2023-12-02T02:46:20.857 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-volume-2:18.0.0-7631.gbed76814.el9.noarch 1/1 2023-12-02T02:46:20.858 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:20.858 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2023-12-02T02:46:20.858 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-volume-2:18.0.0-7631.gbed76814.el9.noarch 2023-12-02T02:46:20.858 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:20.858 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-12-02T02:46:20.866 INFO:teuthology.orchestra.run.smithi029.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 821 kB/s | 34 kB 00:00 2023-12-02T02:46:21.075 INFO:teuthology.orchestra.run.smithi029.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2023-12-02T02:46:21.150 INFO:teuthology.orchestra.run.smithi029.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.6 MB/s | 3.0 MB 00:00 2023-12-02T02:46:21.192 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install librados-devel 2023-12-02T02:46:21.278 INFO:teuthology.orchestra.run.smithi029.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 4.8 MB 00:00 2023-12-02T02:46:21.312 INFO:teuthology.orchestra.run.smithi029.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.3 MB/s | 217 kB 00:00 2023-12-02T02:46:21.471 INFO:teuthology.orchestra.run.smithi029.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 5.0 MB/s | 803 kB 00:00 2023-12-02T02:46:21.796 INFO:teuthology.orchestra.run.smithi029.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2023-12-02T02:46:21.802 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:40 ago on Sat 02 Dec 2023 02:44:41 AM UTC. 2023-12-02T02:46:21.822 INFO:teuthology.orchestra.run.smithi029.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 561 kB/s | 195 kB 00:00 2023-12-02T02:46:21.953 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-12-02T02:46:21.954 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:46:21.954 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repo Size 2023-12-02T02:46:21.954 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:46:21.954 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2023-12-02T02:46:21.954 INFO:teuthology.orchestra.run.smithi057.stdout: librados-devel x86_64 2:18.0.0-7631.gbed76814.el9 ceph 127 k 2023-12-02T02:46:21.954 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:21.954 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2023-12-02T02:46:21.954 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:46:21.954 INFO:teuthology.orchestra.run.smithi057.stdout:Install 1 Package 2023-12-02T02:46:21.954 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:21.954 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 127 k 2023-12-02T02:46:21.954 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 456 k 2023-12-02T02:46:21.955 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2023-12-02T02:46:22.155 INFO:teuthology.orchestra.run.smithi029.stdout:(11/12): ceph-mgr-diskprediction-local-18.0.0-7 4.8 MB/s | 7.4 MB 00:01 2023-12-02T02:46:22.189 INFO:teuthology.orchestra.run.smithi029.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 626 kB/s | 245 kB 00:00 2023-12-02T02:46:22.192 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:46:22.192 INFO:teuthology.orchestra.run.smithi029.stdout:Total 22 MB/s | 37 MB 00:01 2023-12-02T02:46:22.207 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-02T02:46:22.235 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-02T02:46:22.235 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-02T02:46:22.302 INFO:teuthology.orchestra.run.smithi057.stdout:librados-devel-18.0.0-7631.gbed76814.el9.x86_64 365 kB/s | 127 kB 00:00 2023-12-02T02:46:22.302 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:46:22.302 INFO:teuthology.orchestra.run.smithi057.stdout:Total 363 kB/s | 127 kB 00:00 2023-12-02T02:46:22.303 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2023-12-02T02:46:22.309 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2023-12-02T02:46:22.309 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2023-12-02T02:46:22.367 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2023-12-02T02:46:22.368 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2023-12-02T02:46:22.523 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2023-12-02T02:46:22.619 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librados-devel-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:46:22.650 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-02T02:46:22.650 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-02T02:46:23.090 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librados-devel-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:46:23.371 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-02T02:46:23.393 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librados-devel-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:46:23.393 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:23.393 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2023-12-02T02:46:23.393 INFO:teuthology.orchestra.run.smithi057.stdout: librados-devel-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:46:23.393 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:23.393 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-12-02T02:46:23.463 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-02T02:46:23.495 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-02T02:46:23.601 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-02T02:46:23.639 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install libcephfs2 2023-12-02T02:46:23.642 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-02T02:46:23.831 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-02T02:46:23.867 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-02T02:46:23.976 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-02T02:46:24.251 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:43 ago on Sat 02 Dec 2023 02:44:41 AM UTC. 2023-12-02T02:46:24.349 INFO:teuthology.orchestra.run.smithi057.stdout:Package libcephfs2-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T02:46:24.402 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-12-02T02:46:24.403 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2023-12-02T02:46:24.403 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-12-02T02:46:24.465 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install libcephfs-devel 2023-12-02T02:46:24.681 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-02T02:46:24.799 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-02T02:46:25.071 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:44 ago on Sat 02 Dec 2023 02:44:41 AM UTC. 2023-12-02T02:46:25.224 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-12-02T02:46:25.224 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:46:25.225 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repo Size 2023-12-02T02:46:25.225 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:46:25.225 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2023-12-02T02:46:25.225 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs-devel x86_64 2:18.0.0-7631.gbed76814.el9 ceph 31 k 2023-12-02T02:46:25.225 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:25.225 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2023-12-02T02:46:25.225 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:46:25.225 INFO:teuthology.orchestra.run.smithi057.stdout:Install 1 Package 2023-12-02T02:46:25.225 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:25.225 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 31 k 2023-12-02T02:46:25.225 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 139 k 2023-12-02T02:46:25.225 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2023-12-02T02:46:25.463 INFO:teuthology.orchestra.run.smithi057.stdout:libcephfs-devel-18.0.0-7631.gbed76814.el9.x86_6 131 kB/s | 31 kB 00:00 2023-12-02T02:46:25.464 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:46:25.464 INFO:teuthology.orchestra.run.smithi057.stdout:Total 130 kB/s | 31 kB 00:00 2023-12-02T02:46:25.464 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2023-12-02T02:46:25.467 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2023-12-02T02:46:25.467 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2023-12-02T02:46:25.496 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2023-12-02T02:46:25.497 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2023-12-02T02:46:25.590 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2023-12-02T02:46:25.697 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libcephfs-devel-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:46:26.081 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:46:26.123 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-02T02:46:26.367 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libcephfs-devel-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:46:26.368 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:26.368 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2023-12-02T02:46:26.368 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs-devel-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:46:26.368 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:26.368 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-12-02T02:46:26.528 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install librados2 2023-12-02T02:46:27.135 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:46 ago on Sat 02 Dec 2023 02:44:41 AM UTC. 2023-12-02T02:46:27.226 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-02T02:46:27.234 INFO:teuthology.orchestra.run.smithi057.stdout:Package librados2-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T02:46:27.257 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7631.gbed76 12/12 2023-12-02T02:46:27.288 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-12-02T02:46:27.289 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2023-12-02T02:46:27.289 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-12-02T02:46:27.345 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install librbd1 2023-12-02T02:46:27.955 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:46 ago on Sat 02 Dec 2023 02:44:41 AM UTC. 2023-12-02T02:46:28.054 INFO:teuthology.orchestra.run.smithi057.stdout:Package librbd1-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T02:46:28.107 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-12-02T02:46:28.108 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2023-12-02T02:46:28.108 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-12-02T02:46:28.164 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install python3-rados 2023-12-02T02:46:28.770 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:47 ago on Sat 02 Dec 2023 02:44:41 AM UTC. 2023-12-02T02:46:28.869 INFO:teuthology.orchestra.run.smithi057.stdout:Package python3-rados-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T02:46:28.923 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-12-02T02:46:28.924 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2023-12-02T02:46:28.924 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-12-02T02:46:28.979 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install python3-rgw 2023-12-02T02:46:29.584 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:48 ago on Sat 02 Dec 2023 02:44:41 AM UTC. 2023-12-02T02:46:29.683 INFO:teuthology.orchestra.run.smithi057.stdout:Package python3-rgw-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T02:46:29.736 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-12-02T02:46:29.738 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2023-12-02T02:46:29.738 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-12-02T02:46:29.814 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install python3-cephfs 2023-12-02T02:46:30.419 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:49 ago on Sat 02 Dec 2023 02:44:41 AM UTC. 2023-12-02T02:46:30.519 INFO:teuthology.orchestra.run.smithi057.stdout:Package python3-cephfs-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T02:46:30.572 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-12-02T02:46:30.573 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2023-12-02T02:46:30.574 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-12-02T02:46:30.632 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install python3-rbd 2023-12-02T02:46:31.237 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:50 ago on Sat 02 Dec 2023 02:44:41 AM UTC. 2023-12-02T02:46:31.263 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7631.gbed76 12/12 2023-12-02T02:46:31.264 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7631.gbed76 1/12 2023-12-02T02:46:31.264 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-02T02:46:31.264 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-02T02:46:31.264 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-02T02:46:31.264 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-02T02:46:31.264 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-02T02:46:31.264 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-02T02:46:31.264 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-02T02:46:31.264 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-02T02:46:31.264 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-02T02:46:31.264 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-02T02:46:31.336 INFO:teuthology.orchestra.run.smithi057.stdout:Package python3-rbd-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T02:46:31.389 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-12-02T02:46:31.390 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2023-12-02T02:46:31.390 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-12-02T02:46:31.443 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install rbd-fuse 2023-12-02T02:46:31.723 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-02T02:46:31.723 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:31.723 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-02T02:46:31.723 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7631.gbed76814.el9.noarch 2023-12-02T02:46:31.723 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-02T02:46:31.723 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-02T02:46:31.723 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-02T02:46:31.723 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-02T02:46:31.723 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-02T02:46:31.723 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-02T02:46:31.723 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-02T02:46:31.723 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-02T02:46:31.723 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-02T02:46:31.724 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-02T02:46:31.724 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-02T02:46:31.724 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:31.724 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-02T02:46:32.052 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:51 ago on Sat 02 Dec 2023 02:44:41 AM UTC. 2023-12-02T02:46:32.190 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-rook 2023-12-02T02:46:32.203 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-12-02T02:46:32.204 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:46:32.204 INFO:teuthology.orchestra.run.smithi057.stdout: Package Architecture Version Repository Size 2023-12-02T02:46:32.204 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:46:32.204 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2023-12-02T02:46:32.204 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-fuse x86_64 2:18.0.0-7631.gbed76814.el9 ceph 85 k 2023-12-02T02:46:32.204 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:32.204 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2023-12-02T02:46:32.204 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:46:32.205 INFO:teuthology.orchestra.run.smithi057.stdout:Install 1 Package 2023-12-02T02:46:32.205 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:32.205 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 85 k 2023-12-02T02:46:32.205 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 231 k 2023-12-02T02:46:32.205 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2023-12-02T02:46:32.574 INFO:teuthology.orchestra.run.smithi057.stdout:rbd-fuse-18.0.0-7631.gbed76814.el9.x86_64.rpm 231 kB/s | 85 kB 00:00 2023-12-02T02:46:32.575 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:46:32.575 INFO:teuthology.orchestra.run.smithi057.stdout:Total 230 kB/s | 85 kB 00:00 2023-12-02T02:46:32.575 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2023-12-02T02:46:32.583 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2023-12-02T02:46:32.583 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2023-12-02T02:46:32.642 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2023-12-02T02:46:32.643 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2023-12-02T02:46:32.785 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2023-12-02T02:46:32.802 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:44 ago on Sat 02 Dec 2023 02:44:48 AM UTC. 2023-12-02T02:46:32.883 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : rbd-fuse-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:46:32.955 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-02T02:46:32.956 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2023-12-02T02:46:32.956 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-12-02T02:46:32.956 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2023-12-02T02:46:32.956 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-02T02:46:32.957 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook noarch 2:18.0.0-7631.gbed76814.el9 ceph-noarch 50 k 2023-12-02T02:46:32.957 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-12-02T02:46:32.957 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-02T02:46:32.957 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-02T02:46:32.957 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-02T02:46:32.957 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-02T02:46:32.957 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-02T02:46:32.957 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-02T02:46:32.957 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-02T02:46:32.957 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-02T02:46:32.957 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-02T02:46:32.957 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-02T02:46:32.957 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-02T02:46:32.957 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-02T02:46:32.958 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:32.958 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-02T02:46:32.958 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2023-12-02T02:46:32.958 INFO:teuthology.orchestra.run.smithi029.stdout:Install 13 Packages 2023-12-02T02:46:32.958 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:32.958 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 2.0 M 2023-12-02T02:46:32.959 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 26 M 2023-12-02T02:46:32.959 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-02T02:46:33.128 INFO:teuthology.orchestra.run.smithi029.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 203 kB/s | 24 kB 00:00 2023-12-02T02:46:33.145 INFO:teuthology.orchestra.run.smithi029.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 132 kB/s | 18 kB 00:00 2023-12-02T02:46:33.196 INFO:teuthology.orchestra.run.smithi029.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2023-12-02T02:46:33.222 INFO:teuthology.orchestra.run.smithi029.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2023-12-02T02:46:33.248 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: rbd-fuse-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:46:33.256 INFO:teuthology.orchestra.run.smithi029.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2023-12-02T02:46:33.273 INFO:teuthology.orchestra.run.smithi029.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 848 kB/s | 43 kB 00:00 2023-12-02T02:46:33.382 INFO:teuthology.orchestra.run.smithi029.stdout:(7/13): ceph-mgr-rook-18.0.0-7631.gbed76814.el9 134 kB/s | 50 kB 00:00 2023-12-02T02:46:33.426 INFO:teuthology.orchestra.run.smithi029.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 98 kB/s | 15 kB 00:00 2023-12-02T02:46:33.442 INFO:teuthology.orchestra.run.smithi029.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 173 kB/s | 32 kB 00:00 2023-12-02T02:46:33.618 INFO:teuthology.orchestra.run.smithi029.stdout:(10/13): python3-google-auth-2.23.4-1.el9.noarc 836 kB/s | 197 kB 00:00 2023-12-02T02:46:33.625 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : rbd-fuse-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:46:33.625 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:33.625 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2023-12-02T02:46:33.625 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-fuse-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:46:33.625 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:33.625 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-12-02T02:46:33.635 INFO:teuthology.orchestra.run.smithi029.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 307 kB/s | 59 kB 00:00 2023-12-02T02:46:33.819 INFO:teuthology.orchestra.run.smithi029.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 447 kB/s | 90 kB 00:00 2023-12-02T02:46:33.831 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install rbd-mirror 2023-12-02T02:46:33.969 INFO:teuthology.orchestra.run.smithi029.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2023-12-02T02:46:33.974 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:46:33.974 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2023-12-02T02:46:33.974 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-02T02:46:33.987 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-02T02:46:33.987 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-02T02:46:34.095 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-02T02:46:34.096 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-02T02:46:34.444 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:53 ago on Sat 02 Dec 2023 02:44:41 AM UTC. 2023-12-02T02:46:34.596 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-12-02T02:46:34.597 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:46:34.597 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repo Size 2023-12-02T02:46:34.597 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:46:34.597 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2023-12-02T02:46:34.597 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-mirror x86_64 2:18.0.0-7631.gbed76814.el9 ceph 3.0 M 2023-12-02T02:46:34.597 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:34.597 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2023-12-02T02:46:34.597 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:46:34.597 INFO:teuthology.orchestra.run.smithi057.stdout:Install 1 Package 2023-12-02T02:46:34.597 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:34.598 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 3.0 M 2023-12-02T02:46:34.598 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 12 M 2023-12-02T02:46:34.598 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2023-12-02T02:46:34.632 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-02T02:46:34.811 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-02T02:46:34.874 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-02T02:46:34.938 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-02T02:46:34.985 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-02T02:46:35.028 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-02T02:46:35.134 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-02T02:46:35.151 INFO:teuthology.orchestra.run.smithi057.stdout:rbd-mirror-18.0.0-7631.gbed76814.el9.x86_64.rpm 5.4 MB/s | 3.0 MB 00:00 2023-12-02T02:46:35.152 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:46:35.152 INFO:teuthology.orchestra.run.smithi057.stdout:Total 5.4 MB/s | 3.0 MB 00:00 2023-12-02T02:46:35.152 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2023-12-02T02:46:35.163 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2023-12-02T02:46:35.163 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2023-12-02T02:46:35.240 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2023-12-02T02:46:35.240 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2023-12-02T02:46:35.244 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-02T02:46:35.307 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-02T02:46:35.680 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2023-12-02T02:46:35.744 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : rbd-mirror-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:46:35.770 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:46:35.770 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T02:46:35.770 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-02T02:46:35.770 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-02T02:46:35.770 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-02T02:46:35.771 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:36.078 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-02T02:46:36.149 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-02T02:46:36.184 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-02T02:46:36.243 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-02T02:46:36.264 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-rook-2:18.0.0-7631.gbed76814.el9.noarch 13/13 2023-12-02T02:46:36.934 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : rbd-mirror-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:46:36.934 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:36.934 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2023-12-02T02:46:36.934 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-mirror-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:46:36.935 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:36.935 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-12-02T02:46:37.181 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install rbd-nbd 2023-12-02T02:46:37.322 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7631.gbed76814.el9.noarch 13/13 2023-12-02T02:46:37.322 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7631.gbed76814.el9.noarch 1/13 2023-12-02T02:46:37.322 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-02T02:46:37.322 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-02T02:46:37.322 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-02T02:46:37.322 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-02T02:46:37.322 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-02T02:46:37.322 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-02T02:46:37.323 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-02T02:46:37.323 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-02T02:46:37.323 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-02T02:46:37.323 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-02T02:46:37.323 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-02T02:46:37.712 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-02T02:46:37.712 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:37.712 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-02T02:46:37.712 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook-2:18.0.0-7631.gbed76814.el9.noarch 2023-12-02T02:46:37.712 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-02T02:46:37.713 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-02T02:46:37.713 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-02T02:46:37.713 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-02T02:46:37.713 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-02T02:46:37.713 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-02T02:46:37.713 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-02T02:46:37.713 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-02T02:46:37.713 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-02T02:46:37.713 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-02T02:46:37.713 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-02T02:46:37.713 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-02T02:46:37.713 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:37.713 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-02T02:46:37.793 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:56 ago on Sat 02 Dec 2023 02:44:41 AM UTC. 2023-12-02T02:46:37.946 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-12-02T02:46:37.947 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:46:37.947 INFO:teuthology.orchestra.run.smithi057.stdout: Package Architecture Version Repository Size 2023-12-02T02:46:37.947 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:46:37.947 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2023-12-02T02:46:37.947 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-nbd x86_64 2:18.0.0-7631.gbed76814.el9 ceph 170 k 2023-12-02T02:46:37.947 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:37.947 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2023-12-02T02:46:37.947 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-12-02T02:46:37.947 INFO:teuthology.orchestra.run.smithi057.stdout:Install 1 Package 2023-12-02T02:46:37.947 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:37.947 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 170 k 2023-12-02T02:46:37.947 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 494 k 2023-12-02T02:46:37.947 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2023-12-02T02:46:38.060 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-cephadm 2023-12-02T02:46:38.296 INFO:teuthology.orchestra.run.smithi057.stdout:rbd-nbd-18.0.0-7631.gbed76814.el9.x86_64.rpm 489 kB/s | 170 kB 00:00 2023-12-02T02:46:38.296 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:46:38.297 INFO:teuthology.orchestra.run.smithi057.stdout:Total 487 kB/s | 170 kB 00:00 2023-12-02T02:46:38.297 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2023-12-02T02:46:38.305 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2023-12-02T02:46:38.306 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2023-12-02T02:46:38.364 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2023-12-02T02:46:38.365 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2023-12-02T02:46:38.527 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2023-12-02T02:46:38.639 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : rbd-nbd-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:46:38.665 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:50 ago on Sat 02 Dec 2023 02:44:48 AM UTC. 2023-12-02T02:46:38.819 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-02T02:46:38.819 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2023-12-02T02:46:38.819 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-12-02T02:46:38.819 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2023-12-02T02:46:38.820 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-02T02:46:38.820 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7631.gbed76814.el9 ceph-noarch 133 k 2023-12-02T02:46:38.820 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-12-02T02:46:38.820 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-02T02:46:38.820 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-02T02:46:38.820 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-02T02:46:38.820 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-02T02:46:38.820 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-02T02:46:38.820 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:38.820 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-02T02:46:38.820 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2023-12-02T02:46:38.820 INFO:teuthology.orchestra.run.smithi029.stdout:Install 6 Packages 2023-12-02T02:46:38.820 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:38.821 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 6.8 M 2023-12-02T02:46:38.821 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 33 M 2023-12-02T02:46:38.821 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-02T02:46:39.120 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: rbd-nbd-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:46:39.153 INFO:teuthology.orchestra.run.smithi029.stdout:(1/6): ceph-mgr-cephadm-18.0.0-7631.gbed76814.e 527 kB/s | 133 kB 00:00 2023-12-02T02:46:39.270 INFO:teuthology.orchestra.run.smithi029.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2023-12-02T02:46:39.322 INFO:teuthology.orchestra.run.smithi029.stdout:(3/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 545 kB/s | 228 kB 00:00 2023-12-02T02:46:39.451 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : rbd-nbd-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:46:39.451 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:39.451 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2023-12-02T02:46:39.451 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-nbd-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:46:39.451 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:46:39.451 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-12-02T02:46:39.588 INFO:teuthology.orchestra.run.smithi029.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 182 kB/s | 58 kB 00:00 2023-12-02T02:46:39.646 INFO:teuthology.orchestra.run.smithi029.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 184 kB/s | 51 kB 00:00 2023-12-02T02:46:39.679 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install rbd-mirror 2023-12-02T02:46:39.755 INFO:teuthology.orchestra.run.smithi029.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 912 kB/s | 548 kB 00:00 2023-12-02T02:46:39.759 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:46:39.759 INFO:teuthology.orchestra.run.smithi029.stdout:Total 7.2 MB/s | 6.8 MB 00:00 2023-12-02T02:46:39.759 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-02T02:46:39.769 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-02T02:46:39.769 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-02T02:46:39.865 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-02T02:46:39.866 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-02T02:46:40.180 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-02T02:46:40.303 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:59 ago on Sat 02 Dec 2023 02:44:41 AM UTC. 2023-12-02T02:46:40.367 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-02T02:46:40.405 INFO:teuthology.orchestra.run.smithi057.stdout:Package rbd-mirror-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T02:46:40.428 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-02T02:46:40.460 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-12-02T02:46:40.461 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2023-12-02T02:46:40.461 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-12-02T02:46:40.522 DEBUG:teuthology.parallel:result is None 2023-12-02T02:46:40.999 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-02T02:46:41.092 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-02T02:46:41.164 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-02T02:46:41.202 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7631.gbed76814.el9.noarch 6/6 2023-12-02T02:46:42.299 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7631.gbed76814.el9.noarch 6/6 2023-12-02T02:46:42.300 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7631.gbed76814.el9.noarch 1/6 2023-12-02T02:46:42.300 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-02T02:46:42.300 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-02T02:46:42.300 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-02T02:46:42.300 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-02T02:46:42.618 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-02T02:46:42.618 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:42.618 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-02T02:46:42.618 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm-2:18.0.0-7631.gbed76814.el9.noarch 2023-12-02T02:46:42.618 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-02T02:46:42.619 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-02T02:46:42.619 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-02T02:46:42.619 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-02T02:46:42.619 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-02T02:46:42.619 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:42.619 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-02T02:46:42.943 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-fuse 2023-12-02T02:46:43.545 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:55 ago on Sat 02 Dec 2023 02:44:48 AM UTC. 2023-12-02T02:46:43.699 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-02T02:46:43.700 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:46:43.701 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-12-02T02:46:43.701 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:46:43.701 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-02T02:46:43.701 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse x86_64 2:18.0.0-7631.gbed76814.el9 ceph 859 k 2023-12-02T02:46:43.701 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-12-02T02:46:43.701 INFO:teuthology.orchestra.run.smithi029.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-02T02:46:43.701 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:43.702 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-02T02:46:43.702 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:46:43.702 INFO:teuthology.orchestra.run.smithi029.stdout:Install 2 Packages 2023-12-02T02:46:43.702 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:43.702 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 938 k 2023-12-02T02:46:43.702 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 2.7 M 2023-12-02T02:46:43.703 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-02T02:46:43.829 INFO:teuthology.orchestra.run.smithi029.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 617 kB/s | 79 kB 00:00 2023-12-02T02:46:44.029 INFO:teuthology.orchestra.run.smithi029.stdout:(2/2): ceph-fuse-18.0.0-7631.gbed76814.el9.x86_ 2.6 MB/s | 859 kB 00:00 2023-12-02T02:46:44.030 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:46:44.031 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.8 MB/s | 938 kB 00:00 2023-12-02T02:46:44.031 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-02T02:46:44.053 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-02T02:46:44.054 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-02T02:46:44.134 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-02T02:46:44.135 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-02T02:46:44.316 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-02T02:46:44.459 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-02T02:46:44.541 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-fuse-2:18.0.0-7631.gbed76814.el9.x86_64 2/2 2023-12-02T02:46:45.422 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-fuse-2:18.0.0-7631.gbed76814.el9.x86_64 2/2 2023-12-02T02:46:45.422 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-fuse-2:18.0.0-7631.gbed76814.el9.x86_64 1/2 2023-12-02T02:46:45.762 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-02T02:46:45.763 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:45.763 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-02T02:46:45.763 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse-2:18.0.0-7631.gbed76814.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-02T02:46:45.763 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:45.763 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-02T02:46:46.142 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-volume 2023-12-02T02:46:46.751 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:58 ago on Sat 02 Dec 2023 02:44:48 AM UTC. 2023-12-02T02:46:46.902 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-02T02:46:46.903 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:46:46.903 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-12-02T02:46:46.903 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:46:46.903 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-02T02:46:46.903 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume noarch 2:18.0.0-7631.gbed76814.el9 ceph-noarch 262 k 2023-12-02T02:46:46.903 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:46.904 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-02T02:46:46.904 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:46:46.904 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-12-02T02:46:46.904 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:46.904 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 262 k 2023-12-02T02:46:46.904 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 1.2 M 2023-12-02T02:46:46.904 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-02T02:46:47.409 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-volume-18.0.0-7631.gbed76814.el9.noarch.rp 520 kB/s | 262 kB 00:00 2023-12-02T02:46:47.409 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:46:47.409 INFO:teuthology.orchestra.run.smithi029.stdout:Total 519 kB/s | 262 kB 00:00 2023-12-02T02:46:47.410 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-02T02:46:47.418 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-02T02:46:47.419 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-02T02:46:47.461 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-02T02:46:47.461 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-02T02:46:47.688 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-02T02:46:47.770 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-volume-2:18.0.0-7631.gbed76814.el9.noarch 1/1 2023-12-02T02:46:47.787 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-volume-2:18.0.0-7631.gbed76814.el9.noarch 1/1 2023-12-02T02:46:47.787 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T02:46:47.787 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-02T02:46:47.787 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:49.019 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-volume-2:18.0.0-7631.gbed76814.el9.noarch 1/1 2023-12-02T02:46:49.019 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:49.019 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-02T02:46:49.019 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume-2:18.0.0-7631.gbed76814.el9.noarch 2023-12-02T02:46:49.019 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:49.020 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-02T02:46:49.418 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados-devel 2023-12-02T02:46:50.031 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:02 ago on Sat 02 Dec 2023 02:44:48 AM UTC. 2023-12-02T02:46:50.185 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-02T02:46:50.185 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:46:50.185 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-12-02T02:46:50.186 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:46:50.186 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-02T02:46:50.186 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel x86_64 2:18.0.0-7631.gbed76814.el9 ceph 127 k 2023-12-02T02:46:50.186 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:50.186 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-02T02:46:50.186 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:46:50.186 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-12-02T02:46:50.186 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:50.186 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 127 k 2023-12-02T02:46:50.186 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 456 k 2023-12-02T02:46:50.186 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-02T02:46:50.490 INFO:teuthology.orchestra.run.smithi029.stdout:librados-devel-18.0.0-7631.gbed76814.el9.x86_64 418 kB/s | 127 kB 00:00 2023-12-02T02:46:50.490 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:46:50.490 INFO:teuthology.orchestra.run.smithi029.stdout:Total 416 kB/s | 127 kB 00:00 2023-12-02T02:46:50.490 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-02T02:46:50.497 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-02T02:46:50.497 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-02T02:46:50.555 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-02T02:46:50.555 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-02T02:46:50.711 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-02T02:46:50.825 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librados-devel-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:46:51.355 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados-devel-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:46:51.719 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados-devel-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:46:51.719 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:51.719 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-02T02:46:51.719 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:46:51.719 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:51.719 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-02T02:46:52.000 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install libcephfs2 2023-12-02T02:46:52.614 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:04 ago on Sat 02 Dec 2023 02:44:48 AM UTC. 2023-12-02T02:46:52.712 INFO:teuthology.orchestra.run.smithi029.stdout:Package libcephfs2-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T02:46:52.766 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-02T02:46:52.767 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-12-02T02:46:52.767 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-02T02:46:52.853 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install libcephfs-devel 2023-12-02T02:46:53.462 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:05 ago on Sat 02 Dec 2023 02:44:48 AM UTC. 2023-12-02T02:46:53.616 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-02T02:46:53.617 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:46:53.617 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-12-02T02:46:53.617 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:46:53.617 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-02T02:46:53.617 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel x86_64 2:18.0.0-7631.gbed76814.el9 ceph 31 k 2023-12-02T02:46:53.617 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:53.617 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-02T02:46:53.617 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:46:53.617 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-12-02T02:46:53.617 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:53.617 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 31 k 2023-12-02T02:46:53.617 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 139 k 2023-12-02T02:46:53.618 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-02T02:46:54.052 INFO:teuthology.orchestra.run.smithi029.stdout:libcephfs-devel-18.0.0-7631.gbed76814.el9.x86_6 72 kB/s | 31 kB 00:00 2023-12-02T02:46:54.052 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:46:54.053 INFO:teuthology.orchestra.run.smithi029.stdout:Total 71 kB/s | 31 kB 00:00 2023-12-02T02:46:54.053 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-02T02:46:54.056 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-02T02:46:54.056 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-02T02:46:54.093 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-02T02:46:54.093 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-02T02:46:54.185 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-02T02:46:54.320 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs-devel-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:46:54.757 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:46:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs-devel-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:46:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:55.040 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-02T02:46:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:46:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:46:55.040 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-02T02:46:55.276 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados2 2023-12-02T02:46:55.885 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:07 ago on Sat 02 Dec 2023 02:44:48 AM UTC. 2023-12-02T02:46:55.984 INFO:teuthology.orchestra.run.smithi029.stdout:Package librados2-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T02:46:56.038 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-02T02:46:56.039 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-12-02T02:46:56.039 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-02T02:46:56.115 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librbd1 2023-12-02T02:46:56.729 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:08 ago on Sat 02 Dec 2023 02:44:48 AM UTC. 2023-12-02T02:46:56.828 INFO:teuthology.orchestra.run.smithi029.stdout:Package librbd1-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T02:46:56.882 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-02T02:46:56.883 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-12-02T02:46:56.883 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-02T02:46:56.963 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rados 2023-12-02T02:46:57.573 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:09 ago on Sat 02 Dec 2023 02:44:48 AM UTC. 2023-12-02T02:46:57.672 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rados-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T02:46:57.726 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-02T02:46:57.727 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-12-02T02:46:57.727 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-02T02:46:57.808 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rgw 2023-12-02T02:46:58.417 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:10 ago on Sat 02 Dec 2023 02:44:48 AM UTC. 2023-12-02T02:46:58.515 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rgw-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T02:46:58.569 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-02T02:46:58.570 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-12-02T02:46:58.570 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-02T02:46:58.650 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-cephfs 2023-12-02T02:46:59.254 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:11 ago on Sat 02 Dec 2023 02:44:48 AM UTC. 2023-12-02T02:46:59.354 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-cephfs-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T02:46:59.410 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-02T02:46:59.410 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-12-02T02:46:59.410 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-02T02:46:59.491 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rbd 2023-12-02T02:47:00.091 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:12 ago on Sat 02 Dec 2023 02:44:48 AM UTC. 2023-12-02T02:47:00.189 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rbd-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T02:47:00.242 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-02T02:47:00.243 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-12-02T02:47:00.243 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-02T02:47:00.323 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-fuse 2023-12-02T02:47:00.932 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:12 ago on Sat 02 Dec 2023 02:44:48 AM UTC. 2023-12-02T02:47:01.085 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-02T02:47:01.085 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:47:01.085 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2023-12-02T02:47:01.085 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:47:01.085 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-02T02:47:01.085 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse x86_64 2:18.0.0-7631.gbed76814.el9 ceph 85 k 2023-12-02T02:47:01.086 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:01.086 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-02T02:47:01.086 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:47:01.086 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-12-02T02:47:01.086 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:01.086 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 85 k 2023-12-02T02:47:01.086 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 231 k 2023-12-02T02:47:01.087 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-02T02:47:01.302 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-fuse-18.0.0-7631.gbed76814.el9.x86_64.rpm 396 kB/s | 85 kB 00:00 2023-12-02T02:47:01.303 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:47:01.303 INFO:teuthology.orchestra.run.smithi029.stdout:Total 393 kB/s | 85 kB 00:00 2023-12-02T02:47:01.303 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-02T02:47:01.311 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-02T02:47:01.311 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-02T02:47:01.369 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-02T02:47:01.369 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-02T02:47:01.506 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-02T02:47:01.612 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-fuse-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:47:02.035 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-fuse-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:47:02.364 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-fuse-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:47:02.364 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:02.364 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-02T02:47:02.364 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:47:02.364 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:02.364 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-02T02:47:02.671 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-mirror 2023-12-02T02:47:03.282 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:15 ago on Sat 02 Dec 2023 02:44:48 AM UTC. 2023-12-02T02:47:03.434 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-02T02:47:03.435 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:47:03.435 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-12-02T02:47:03.435 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:47:03.435 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-02T02:47:03.435 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror x86_64 2:18.0.0-7631.gbed76814.el9 ceph 3.0 M 2023-12-02T02:47:03.435 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:03.435 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-02T02:47:03.435 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:47:03.435 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-12-02T02:47:03.435 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:03.435 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 3.0 M 2023-12-02T02:47:03.436 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 12 M 2023-12-02T02:47:03.436 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-02T02:47:04.045 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-mirror-18.0.0-7631.gbed76814.el9.x86_64.rpm 4.9 MB/s | 3.0 MB 00:00 2023-12-02T02:47:04.046 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:47:04.046 INFO:teuthology.orchestra.run.smithi029.stdout:Total 4.9 MB/s | 3.0 MB 00:00 2023-12-02T02:47:04.046 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-02T02:47:04.056 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-02T02:47:04.057 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-02T02:47:04.133 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-02T02:47:04.133 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-02T02:47:04.577 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-02T02:47:04.647 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-mirror-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:47:04.672 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:47:04.673 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-02T02:47:04.673 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-02T02:47:04.673 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-02T02:47:04.673 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-02T02:47:04.673 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:05.942 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-mirror-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:47:05.942 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:05.942 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-02T02:47:05.942 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:47:05.942 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:05.942 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-02T02:47:06.256 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-nbd 2023-12-02T02:47:06.872 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:18 ago on Sat 02 Dec 2023 02:44:48 AM UTC. 2023-12-02T02:47:07.023 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-02T02:47:07.024 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:47:07.024 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2023-12-02T02:47:07.024 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:47:07.024 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-12-02T02:47:07.024 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd x86_64 2:18.0.0-7631.gbed76814.el9 ceph 170 k 2023-12-02T02:47:07.024 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:07.024 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-12-02T02:47:07.024 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-12-02T02:47:07.024 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-12-02T02:47:07.024 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:07.025 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 170 k 2023-12-02T02:47:07.025 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 494 k 2023-12-02T02:47:07.025 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-12-02T02:47:07.246 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-nbd-18.0.0-7631.gbed76814.el9.x86_64.rpm 771 kB/s | 170 kB 00:00 2023-12-02T02:47:07.247 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-12-02T02:47:07.247 INFO:teuthology.orchestra.run.smithi029.stdout:Total 765 kB/s | 170 kB 00:00 2023-12-02T02:47:07.247 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-12-02T02:47:07.256 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-12-02T02:47:07.256 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-12-02T02:47:07.314 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-12-02T02:47:07.314 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-12-02T02:47:07.466 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-12-02T02:47:07.592 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-nbd-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:47:08.079 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-nbd-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:47:08.414 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-nbd-2:18.0.0-7631.gbed76814.el9.x86_64 1/1 2023-12-02T02:47:08.414 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:08.414 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-12-02T02:47:08.414 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd-2:18.0.0-7631.gbed76814.el9.x86_64 2023-12-02T02:47:08.414 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:08.414 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-02T02:47:08.718 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-mirror 2023-12-02T02:47:09.334 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:21 ago on Sat 02 Dec 2023 02:44:48 AM UTC. 2023-12-02T02:47:09.433 INFO:teuthology.orchestra.run.smithi029.stdout:Package rbd-mirror-2:18.0.0-7631.gbed76814.el9.x86_64 is already installed. 2023-12-02T02:47:09.486 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-12-02T02:47:09.487 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-12-02T02:47:09.487 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-12-02T02:47:09.572 DEBUG:teuthology.parallel:result is None 2023-12-02T02:47:09.573 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bed7681480ccb0192b91a35e691658867e1e96b7 2023-12-02T02:47:09.722 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-02T02:47:09.744 INFO:teuthology.orchestra.run.smithi029.stdout:18.0.0-7631.gbed76814.el9 2023-12-02T02:47:09.745 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7631.gbed76814.el9 2023-12-02T02:47:09.745 INFO:teuthology.task.install:The correct ceph version 18.0.0-7631.gbed76814 is installed. 2023-12-02T02:47:09.746 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bed7681480ccb0192b91a35e691658867e1e96b7 2023-12-02T02:47:09.955 DEBUG:teuthology.orchestra.run.smithi057:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-02T02:47:09.980 INFO:teuthology.orchestra.run.smithi057.stdout:18.0.0-7631.gbed76814.el9 2023-12-02T02:47:09.981 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7631.gbed76814.el9 2023-12-02T02:47:09.981 INFO:teuthology.task.install:The correct ceph version 18.0.0-7631.gbed76814 is installed. 2023-12-02T02:47:09.981 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-02T02:47:09.981 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-02T02:47:09.981 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-02T02:47:10.014 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:47:10.014 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-02T02:47:10.056 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-02T02:47:10.057 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-02T02:47:10.057 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/daemon-helper 2023-12-02T02:47:10.088 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-02T02:47:10.159 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:47:10.159 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/daemon-helper 2023-12-02T02:47:10.195 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-02T02:47:10.270 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-02T02:47:10.270 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-02T02:47:10.270 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-02T02:47:10.302 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-02T02:47:10.374 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:47:10.374 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-02T02:47:10.409 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-02T02:47:10.481 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-02T02:47:10.481 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-02T02:47:10.482 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/stdin-killer 2023-12-02T02:47:10.513 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-02T02:47:10.584 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:47:10.584 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/stdin-killer 2023-12-02T02:47:10.615 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-02T02:47:10.687 INFO:teuthology.run_tasks:Running task ceph... 2023-12-02T02:47:10.760 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-02T02:47:10.760 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 777 /var/log/ceph 2023-12-02T02:47:10.762 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 777 /var/log/ceph 2023-12-02T02:47:10.796 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-02T02:47:10.796 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-02T02:47:10.834 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-02T02:47:10.866 INFO:tasks.ceph:Creating extra log directories... 2023-12-02T02:47:10.867 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-02T02:47:10.904 DEBUG:teuthology.orchestra.run.smithi057:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-02T02:47:10.953 INFO:tasks.ceph:Creating ceph cluster cluster1... 2023-12-02T02:47:10.953 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-02T02:47:10.953 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-12-01_18:01:25-rbd-main-distro-default-smithi/7474803', 'branch': 'main', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7474803', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'yuriw-2023-12-01_18:01:25-rbd-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'bed7681480ccb0192b91a35e691658867e1e96b7'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'bed7681480ccb0192b91a35e691658867e1e96b7'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': 'bed7681480ccb0192b91a35e691658867e1e96b7'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 2451, 'sha1': 'bed7681480ccb0192b91a35e691658867e1e96b7', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'bed7681480ccb0192b91a35e691658867e1e96b7', 'targets': {'smithi029.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHhvEccHXaj9KecRdhJ2R7k6deK/FnDjkxOTwaL94Ze0GkNEzZbozrPwNc2SMmJZq3gdDq6vTnghyy+IyRAxjQw=', 'smithi057.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBcE1vhLAy0ZftcWRW+Iu01zQaKILaNE+BFn/TIqb3zF05SMgiwusZ/3kJVVvXngHrx6SL2rEFn+kkOfgIxp3HY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'bed7681480ccb0192b91a35e691658867e1e96b7'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'bed7681480ccb0192b91a35e691658867e1e96b7'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'f2de108beb0a6dde7f3e6c96039071828a1f6269', 'timestamp': '2023-12-01_18:01:25', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-02T02:47:10.954 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-12-02T02:47:10.975 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-12-02T02:47:10.993 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m0777 -- /var/run/ceph 2023-12-02T02:47:11.031 DEBUG:teuthology.orchestra.run.smithi057:> sudo install -d -m0777 -- /var/run/ceph 2023-12-02T02:47:11.064 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-02T02:47:11.064 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2023-12-02T02:47:11.114 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-02T02:47:11.115 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_1 2023-12-02T02:47:11.169 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-02T02:47:11.169 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-02T02:47:11.169 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 662 Links: 1 2023-12-02T02:47:11.170 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-02T02:47:11.170 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2023-12-02T02:47:11.170 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2023-12-02 02:47:04.912044001 +0000 2023-12-02T02:47:11.170 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2023-12-02 02:43:48.948016634 +0000 2023-12-02T02:47:11.170 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2023-12-02 02:43:48.948016634 +0000 2023-12-02T02:47:11.170 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2023-12-02 02:43:48.948016634 +0000 2023-12-02T02:47:11.170 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-02T02:47:11.238 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2023-12-02T02:47:11.238 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2023-12-02T02:47:11.239 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 9.4473e-05 s, 5.4 MB/s 2023-12-02T02:47:11.240 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-02T02:47:11.296 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_2 2023-12-02T02:47:11.351 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-02T02:47:11.351 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-02T02:47:11.351 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 684 Links: 1 2023-12-02T02:47:11.351 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-02T02:47:11.352 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2023-12-02T02:47:11.352 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2023-12-02 02:47:04.912044001 +0000 2023-12-02T02:47:11.352 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2023-12-02 02:43:49.491007331 +0000 2023-12-02T02:47:11.352 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2023-12-02 02:43:49.491007331 +0000 2023-12-02T02:47:11.352 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2023-12-02 02:43:49.491007331 +0000 2023-12-02T02:47:11.352 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-02T02:47:11.420 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2023-12-02T02:47:11.420 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2023-12-02T02:47:11.421 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 8.1614e-05 s, 6.3 MB/s 2023-12-02T02:47:11.421 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-02T02:47:11.478 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_3 2023-12-02T02:47:11.533 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-02T02:47:11.533 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-02T02:47:11.533 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 703 Links: 1 2023-12-02T02:47:11.533 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-02T02:47:11.534 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2023-12-02T02:47:11.534 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2023-12-02 02:47:04.913043984 +0000 2023-12-02T02:47:11.534 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2023-12-02 02:43:49.997998645 +0000 2023-12-02T02:47:11.534 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2023-12-02 02:43:49.997998645 +0000 2023-12-02T02:47:11.534 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2023-12-02 02:43:49.997998645 +0000 2023-12-02T02:47:11.534 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-02T02:47:11.602 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2023-12-02T02:47:11.602 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2023-12-02T02:47:11.602 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 8.7442e-05 s, 5.9 MB/s 2023-12-02T02:47:11.603 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-02T02:47:11.659 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_4 2023-12-02T02:47:11.714 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-02T02:47:11.714 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-02T02:47:11.714 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 722 Links: 1 2023-12-02T02:47:11.714 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-02T02:47:11.714 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2023-12-02T02:47:11.715 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2023-12-02 02:47:04.913043984 +0000 2023-12-02T02:47:11.715 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2023-12-02 02:43:50.502989993 +0000 2023-12-02T02:47:11.715 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2023-12-02 02:43:50.502989993 +0000 2023-12-02T02:47:11.715 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2023-12-02 02:43:50.502989993 +0000 2023-12-02T02:47:11.715 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-02T02:47:11.783 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2023-12-02T02:47:11.783 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2023-12-02T02:47:11.783 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 9.7643e-05 s, 5.2 MB/s 2023-12-02T02:47:11.784 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-02T02:47:11.840 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-02T02:47:11.840 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi029.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-02T02:47:11.840 INFO:tasks.ceph:Generating config... 2023-12-02T02:47:11.841 INFO:tasks.ceph:[client] debug rbd = 20 2023-12-02T02:47:11.841 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-12-02T02:47:11.842 INFO:tasks.ceph:[client] log to stderr = False 2023-12-02T02:47:11.842 INFO:tasks.ceph:[client] rbd default features = 125 2023-12-02T02:47:11.842 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2023-12-02T02:47:11.842 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T02:47:11.842 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T02:47:11.842 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T02:47:11.842 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T02:47:11.842 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T02:47:11.842 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T02:47:11.842 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T02:47:11.842 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T02:47:11.842 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-02T02:47:11.842 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-12-02T02:47:11.842 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-02T02:47:11.843 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-02T02:47:11.843 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-02T02:47:11.843 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-02T02:47:11.843 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-02T02:47:11.843 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-02T02:47:11.843 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2023-12-02T02:47:11.843 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2023-12-02T02:47:11.843 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-02T02:47:11.843 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-02T02:47:11.843 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-02T02:47:11.843 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-02T02:47:11.843 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-02T02:47:11.843 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-02T02:47:11.843 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-02T02:47:11.843 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-02T02:47:11.844 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-02T02:47:11.844 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-02T02:47:11.844 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-02T02:47:11.844 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-02T02:47:11.844 INFO:tasks.ceph:Setting up cluster1.mon.a... 2023-12-02T02:47:11.844 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2023-12-02T02:47:11.941 INFO:teuthology.orchestra.run.smithi029.stdout:creating /etc/ceph/cluster1.keyring 2023-12-02T02:47:11.943 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2023-12-02T02:47:11.991 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-02T02:47:12.021 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.29')] 2023-12-02T02:47:12.022 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.29', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-12-02T02:47:12.022 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-02T02:47:12.022 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-02T02:47:12.076 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.29 --print /home/ubuntu/cephtest/cluster1.monmap 2023-12-02T02:47:12.155 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2023-12-02T02:47:12.155 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: generated fsid 69e9688d-4570-4fa3-a52d-d6c0eaea7d57 2023-12-02T02:47:12.155 INFO:teuthology.orchestra.run.smithi029.stdout:setting min_mon_release = pacific 2023-12-02T02:47:12.155 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 0 2023-12-02T02:47:12.155 INFO:teuthology.orchestra.run.smithi029.stdout:fsid 69e9688d-4570-4fa3-a52d-d6c0eaea7d57 2023-12-02T02:47:12.156 INFO:teuthology.orchestra.run.smithi029.stdout:last_changed 2023-12-02T02:47:12.153359+0000 2023-12-02T02:47:12.156 INFO:teuthology.orchestra.run.smithi029.stdout:created 2023-12-02T02:47:12.153359+0000 2023-12-02T02:47:12.156 INFO:teuthology.orchestra.run.smithi029.stdout:min_mon_release 16 (pacific) 2023-12-02T02:47:12.156 INFO:teuthology.orchestra.run.smithi029.stdout:election_strategy: 1 2023-12-02T02:47:12.156 INFO:teuthology.orchestra.run.smithi029.stdout:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-12-02T02:47:12.156 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2023-12-02T02:47:12.157 DEBUG:teuthology.orchestra.run.smithi029:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-02T02:47:12.212 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 69e9688d-4570-4fa3-a52d-d6c0eaea7d57... 2023-12-02T02:47:12.213 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-12-02T02:47:12.254 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-12-02T02:47:12.313 INFO:teuthology.orchestra.run.smithi029.stdout:[global] 2023-12-02T02:47:12.313 INFO:teuthology.orchestra.run.smithi029.stdout: chdir = "" 2023-12-02T02:47:12.313 INFO:teuthology.orchestra.run.smithi029.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-02T02:47:12.313 INFO:teuthology.orchestra.run.smithi029.stdout: auth supported = cephx 2023-12-02T02:47:12.313 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.313 INFO:teuthology.orchestra.run.smithi029.stdout: filestore xattr use omap = true 2023-12-02T02:47:12.313 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.313 INFO:teuthology.orchestra.run.smithi029.stdout: mon clock drift allowed = 1.000 2023-12-02T02:47:12.313 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.313 INFO:teuthology.orchestra.run.smithi029.stdout: osd crush chooseleaf type = 0 2023-12-02T02:47:12.314 INFO:teuthology.orchestra.run.smithi029.stdout: auth debug = true 2023-12-02T02:47:12.314 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.314 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on old message = true 2023-12-02T02:47:12.314 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on bug = true 2023-12-02T02:47:12.314 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.314 INFO:teuthology.orchestra.run.smithi029.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-02T02:47:12.314 INFO:teuthology.orchestra.run.smithi029.stdout: mon pg warn max object skew = 0 2023-12-02T02:47:12.314 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.314 INFO:teuthology.orchestra.run.smithi029.stdout: # disable pg_autoscaler by default for new pools 2023-12-02T02:47:12.314 INFO:teuthology.orchestra.run.smithi029.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-02T02:47:12.314 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.314 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default size = 2 2023-12-02T02:47:12.314 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.314 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd allow primary affinity = true 2023-12-02T02:47:12.315 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd allow pg remap = true 2023-12-02T02:47:12.315 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on legacy crush tunables = false 2023-12-02T02:47:12.315 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on crush straw calc version zero = false 2023-12-02T02:47:12.315 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on no sortbitwise = false 2023-12-02T02:47:12.315 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on osd down out interval zero = false 2023-12-02T02:47:12.315 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on too few osds = false 2023-12-02T02:47:12.315 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-02T02:47:12.315 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-02T02:47:12.315 INFO:teuthology.orchestra.run.smithi029.stdout: mon_allow_pool_size_one = true 2023-12-02T02:47:12.315 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.315 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-02T02:47:12.315 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.315 INFO:teuthology.orchestra.run.smithi029.stdout: osd default data pool replay window = 5 2023-12-02T02:47:12.315 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.315 INFO:teuthology.orchestra.run.smithi029.stdout: mon allow pool delete = true 2023-12-02T02:47:12.316 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.316 INFO:teuthology.orchestra.run.smithi029.stdout: mon cluster log file level = debug 2023-12-02T02:47:12.316 INFO:teuthology.orchestra.run.smithi029.stdout: debug asserts on shutdown = true 2023-12-02T02:47:12.316 INFO:teuthology.orchestra.run.smithi029.stdout: mon health detail to clog = false 2023-12-02T02:47:12.316 INFO:teuthology.orchestra.run.smithi029.stdout: mon host = 172.21.15.29 2023-12-02T02:47:12.316 INFO:teuthology.orchestra.run.smithi029.stdout: mon client directed command retry = 5 2023-12-02T02:47:12.316 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on pool no app = False 2023-12-02T02:47:12.316 INFO:teuthology.orchestra.run.smithi029.stdout: ms inject socket failures = 5000 2023-12-02T02:47:12.316 INFO:teuthology.orchestra.run.smithi029.stdout: fsid = 69e9688d-4570-4fa3-a52d-d6c0eaea7d57 2023-12-02T02:47:12.316 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.316 INFO:teuthology.orchestra.run.smithi029.stdout:[osd] 2023-12-02T02:47:12.316 INFO:teuthology.orchestra.run.smithi029.stdout: osd journal size = 100 2023-12-02T02:47:12.316 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.316 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub load threshold = 5.0 2023-12-02T02:47:12.317 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub max interval = 600 2023-12-02T02:47:12.317 INFO:teuthology.orchestra.run.smithi029.stdout: osd mclock profile = high_recovery_ops 2023-12-02T02:47:12.317 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.317 INFO:teuthology.orchestra.run.smithi029.stdout: osd recover clone overlap = true 2023-12-02T02:47:12.317 INFO:teuthology.orchestra.run.smithi029.stdout: osd recovery max chunk = 1048576 2023-12-02T02:47:12.317 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.317 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug shutdown = true 2023-12-02T02:47:12.317 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug op order = true 2023-12-02T02:47:12.317 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug verify stray on activate = true 2023-12-02T02:47:12.317 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug trim objects = true 2023-12-02T02:47:12.317 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.317 INFO:teuthology.orchestra.run.smithi029.stdout: osd open classes on start = true 2023-12-02T02:47:12.317 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug pg log writeout = true 2023-12-02T02:47:12.317 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.318 INFO:teuthology.orchestra.run.smithi029.stdout: osd deep scrub update digest min age = 30 2023-12-02T02:47:12.318 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.318 INFO:teuthology.orchestra.run.smithi029.stdout: osd map max advance = 10 2023-12-02T02:47:12.318 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.318 INFO:teuthology.orchestra.run.smithi029.stdout: journal zero on create = true 2023-12-02T02:47:12.318 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.318 INFO:teuthology.orchestra.run.smithi029.stdout: filestore ondisk finisher threads = 3 2023-12-02T02:47:12.318 INFO:teuthology.orchestra.run.smithi029.stdout: filestore apply finisher threads = 3 2023-12-02T02:47:12.318 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.318 INFO:teuthology.orchestra.run.smithi029.stdout: bdev debug aio = true 2023-12-02T02:47:12.318 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug misdirected ops = true 2023-12-02T02:47:12.318 INFO:teuthology.orchestra.run.smithi029.stdout: bluefs allocator = hybrid 2023-12-02T02:47:12.318 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore allocator = hybrid 2023-12-02T02:47:12.318 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore block size = 96636764160 2023-12-02T02:47:12.319 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore fsck on mount = True 2023-12-02T02:47:12.319 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluefs = 1/20 2023-12-02T02:47:12.319 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluestore = 1/20 2023-12-02T02:47:12.319 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-12-02T02:47:12.319 INFO:teuthology.orchestra.run.smithi029.stdout: debug osd = 20 2023-12-02T02:47:12.319 INFO:teuthology.orchestra.run.smithi029.stdout: debug rocksdb = 4/10 2023-12-02T02:47:12.319 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd backfillfull_ratio = 0.85 2023-12-02T02:47:12.319 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd full ratio = 0.9 2023-12-02T02:47:12.319 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd nearfull ratio = 0.8 2023-12-02T02:47:12.319 INFO:teuthology.orchestra.run.smithi029.stdout: osd failsafe full ratio = 0.95 2023-12-02T02:47:12.320 INFO:teuthology.orchestra.run.smithi029.stdout: osd objectstore = bluestore 2023-12-02T02:47:12.320 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.320 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr] 2023-12-02T02:47:12.320 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-12-02T02:47:12.320 INFO:teuthology.orchestra.run.smithi029.stdout: debug mgr = 20 2023-12-02T02:47:12.320 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 20 2023-12-02T02:47:12.320 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 20 2023-12-02T02:47:12.320 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min pgs per osd = 4 2023-12-02T02:47:12.320 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min bytes per osd = 10 2023-12-02T02:47:12.320 INFO:teuthology.orchestra.run.smithi029.stdout: mgr/telemetry/nag = false 2023-12-02T02:47:12.320 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.320 INFO:teuthology.orchestra.run.smithi029.stdout:[mon] 2023-12-02T02:47:12.320 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-12-02T02:47:12.320 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 20 2023-12-02T02:47:12.321 INFO:teuthology.orchestra.run.smithi029.stdout: debug paxos = 20 2023-12-02T02:47:12.321 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 20 2023-12-02T02:47:12.321 INFO:teuthology.orchestra.run.smithi029.stdout: mon data avail warn = 5 2023-12-02T02:47:12.321 INFO:teuthology.orchestra.run.smithi029.stdout: mon mgr mkfs grace = 240 2023-12-02T02:47:12.321 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min pgs per osd = 4 2023-12-02T02:47:12.321 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd reporter subtree level = osd 2023-12-02T02:47:12.321 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd prime pg temp = true 2023-12-02T02:47:12.321 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min bytes per osd = 10 2023-12-02T02:47:12.321 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.321 INFO:teuthology.orchestra.run.smithi029.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-02T02:47:12.321 INFO:teuthology.orchestra.run.smithi029.stdout: auth mon ticket ttl = 660 # 11m 2023-12-02T02:47:12.321 INFO:teuthology.orchestra.run.smithi029.stdout: auth service ticket ttl = 240 # 4m 2023-12-02T02:47:12.321 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.321 INFO:teuthology.orchestra.run.smithi029.stdout: # don't complain about insecure global_id in the test suite 2023-12-02T02:47:12.321 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-02T02:47:12.322 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-02T02:47:12.322 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.322 INFO:teuthology.orchestra.run.smithi029.stdout: # 1m isn't quite enough 2023-12-02T02:47:12.322 INFO:teuthology.orchestra.run.smithi029.stdout: mon_down_mkfs_grace = 2m 2023-12-02T02:47:12.322 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.322 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_filestore_osds = false 2023-12-02T02:47:12.322 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:12.322 INFO:teuthology.orchestra.run.smithi029.stdout:[client] 2023-12-02T02:47:12.322 INFO:teuthology.orchestra.run.smithi029.stdout: rgw cache enabled = true 2023-12-02T02:47:12.322 INFO:teuthology.orchestra.run.smithi029.stdout: rgw enable ops log = true 2023-12-02T02:47:12.322 INFO:teuthology.orchestra.run.smithi029.stdout: rgw enable usage log = true 2023-12-02T02:47:12.322 INFO:teuthology.orchestra.run.smithi029.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-02T02:47:12.322 INFO:teuthology.orchestra.run.smithi029.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-02T02:47:12.322 INFO:teuthology.orchestra.run.smithi029.stdout: debug rbd = 20 2023-12-02T02:47:12.323 INFO:teuthology.orchestra.run.smithi029.stdout: debug rbd_mirror = 15 2023-12-02T02:47:12.323 INFO:teuthology.orchestra.run.smithi029.stdout: log to stderr = False 2023-12-02T02:47:12.323 INFO:teuthology.orchestra.run.smithi029.stdout: rbd default features = 125 2023-12-02T02:47:12.323 INFO:teuthology.orchestra.run.smithi029.stdout: rbd mirror image policy type = simple 2023-12-02T02:47:12.323 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.a] 2023-12-02T02:47:12.323 INFO:teuthology.orchestra.run.smithi029.stdout:[client.mirror.0] 2023-12-02T02:47:12.323 INFO:teuthology.orchestra.run.smithi029.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T02:47:12.323 INFO:teuthology.orchestra.run.smithi029.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T02:47:12.323 INFO:teuthology.orchestra.run.smithi029.stdout:[client.mirror.1] 2023-12-02T02:47:12.323 INFO:teuthology.orchestra.run.smithi029.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T02:47:12.323 INFO:teuthology.orchestra.run.smithi029.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T02:47:12.323 INFO:teuthology.orchestra.run.smithi029.stdout:[client.mirror.2] 2023-12-02T02:47:12.323 INFO:teuthology.orchestra.run.smithi029.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T02:47:12.323 INFO:teuthology.orchestra.run.smithi029.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T02:47:12.324 INFO:teuthology.orchestra.run.smithi029.stdout:[client.mirror.3] 2023-12-02T02:47:12.324 INFO:teuthology.orchestra.run.smithi029.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T02:47:12.324 INFO:teuthology.orchestra.run.smithi029.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T02:47:12.324 INFO:teuthology.orchestra.run.smithi057.stdout:[global] 2023-12-02T02:47:12.324 INFO:teuthology.orchestra.run.smithi057.stdout: chdir = "" 2023-12-02T02:47:12.324 INFO:teuthology.orchestra.run.smithi057.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-02T02:47:12.324 INFO:teuthology.orchestra.run.smithi057.stdout: auth supported = cephx 2023-12-02T02:47:12.324 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.325 INFO:teuthology.orchestra.run.smithi057.stdout: filestore xattr use omap = true 2023-12-02T02:47:12.325 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.325 INFO:teuthology.orchestra.run.smithi057.stdout: mon clock drift allowed = 1.000 2023-12-02T02:47:12.325 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.325 INFO:teuthology.orchestra.run.smithi057.stdout: osd crush chooseleaf type = 0 2023-12-02T02:47:12.325 INFO:teuthology.orchestra.run.smithi057.stdout: auth debug = true 2023-12-02T02:47:12.325 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.325 INFO:teuthology.orchestra.run.smithi057.stdout: ms die on old message = true 2023-12-02T02:47:12.325 INFO:teuthology.orchestra.run.smithi057.stdout: ms die on bug = true 2023-12-02T02:47:12.325 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.325 INFO:teuthology.orchestra.run.smithi057.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-02T02:47:12.325 INFO:teuthology.orchestra.run.smithi057.stdout: mon pg warn max object skew = 0 2023-12-02T02:47:12.325 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.326 INFO:teuthology.orchestra.run.smithi057.stdout: # disable pg_autoscaler by default for new pools 2023-12-02T02:47:12.326 INFO:teuthology.orchestra.run.smithi057.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-02T02:47:12.326 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.326 INFO:teuthology.orchestra.run.smithi057.stdout: osd pool default size = 2 2023-12-02T02:47:12.326 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.326 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd allow primary affinity = true 2023-12-02T02:47:12.326 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd allow pg remap = true 2023-12-02T02:47:12.326 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on legacy crush tunables = false 2023-12-02T02:47:12.326 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on crush straw calc version zero = false 2023-12-02T02:47:12.326 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on no sortbitwise = false 2023-12-02T02:47:12.326 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on osd down out interval zero = false 2023-12-02T02:47:12.326 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on too few osds = false 2023-12-02T02:47:12.326 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-02T02:47:12.326 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-02T02:47:12.327 INFO:teuthology.orchestra.run.smithi057.stdout: mon_allow_pool_size_one = true 2023-12-02T02:47:12.327 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.327 INFO:teuthology.orchestra.run.smithi057.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-02T02:47:12.327 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.327 INFO:teuthology.orchestra.run.smithi057.stdout: osd default data pool replay window = 5 2023-12-02T02:47:12.327 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.327 INFO:teuthology.orchestra.run.smithi057.stdout: mon allow pool delete = true 2023-12-02T02:47:12.327 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.327 INFO:teuthology.orchestra.run.smithi057.stdout: mon cluster log file level = debug 2023-12-02T02:47:12.327 INFO:teuthology.orchestra.run.smithi057.stdout: debug asserts on shutdown = true 2023-12-02T02:47:12.327 INFO:teuthology.orchestra.run.smithi057.stdout: mon health detail to clog = false 2023-12-02T02:47:12.327 INFO:teuthology.orchestra.run.smithi057.stdout: mon host = 172.21.15.29 2023-12-02T02:47:12.327 INFO:teuthology.orchestra.run.smithi057.stdout: mon client directed command retry = 5 2023-12-02T02:47:12.327 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on pool no app = False 2023-12-02T02:47:12.327 INFO:teuthology.orchestra.run.smithi057.stdout: ms inject socket failures = 5000 2023-12-02T02:47:12.328 INFO:teuthology.orchestra.run.smithi057.stdout: fsid = 69e9688d-4570-4fa3-a52d-d6c0eaea7d57 2023-12-02T02:47:12.328 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.328 INFO:teuthology.orchestra.run.smithi057.stdout:[osd] 2023-12-02T02:47:12.328 INFO:teuthology.orchestra.run.smithi057.stdout: osd journal size = 100 2023-12-02T02:47:12.328 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.328 INFO:teuthology.orchestra.run.smithi057.stdout: osd scrub load threshold = 5.0 2023-12-02T02:47:12.328 INFO:teuthology.orchestra.run.smithi057.stdout: osd scrub max interval = 600 2023-12-02T02:47:12.328 INFO:teuthology.orchestra.run.smithi057.stdout: osd mclock profile = high_recovery_ops 2023-12-02T02:47:12.328 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.328 INFO:teuthology.orchestra.run.smithi057.stdout: osd recover clone overlap = true 2023-12-02T02:47:12.328 INFO:teuthology.orchestra.run.smithi057.stdout: osd recovery max chunk = 1048576 2023-12-02T02:47:12.328 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.328 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug shutdown = true 2023-12-02T02:47:12.328 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug op order = true 2023-12-02T02:47:12.328 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug verify stray on activate = true 2023-12-02T02:47:12.329 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug trim objects = true 2023-12-02T02:47:12.329 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.329 INFO:teuthology.orchestra.run.smithi057.stdout: osd open classes on start = true 2023-12-02T02:47:12.329 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug pg log writeout = true 2023-12-02T02:47:12.329 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.329 INFO:teuthology.orchestra.run.smithi057.stdout: osd deep scrub update digest min age = 30 2023-12-02T02:47:12.329 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.329 INFO:teuthology.orchestra.run.smithi057.stdout: osd map max advance = 10 2023-12-02T02:47:12.329 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.329 INFO:teuthology.orchestra.run.smithi057.stdout: journal zero on create = true 2023-12-02T02:47:12.329 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.329 INFO:teuthology.orchestra.run.smithi057.stdout: filestore ondisk finisher threads = 3 2023-12-02T02:47:12.329 INFO:teuthology.orchestra.run.smithi057.stdout: filestore apply finisher threads = 3 2023-12-02T02:47:12.330 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.330 INFO:teuthology.orchestra.run.smithi057.stdout: bdev debug aio = true 2023-12-02T02:47:12.330 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug misdirected ops = true 2023-12-02T02:47:12.330 INFO:teuthology.orchestra.run.smithi057.stdout: bluefs allocator = hybrid 2023-12-02T02:47:12.330 INFO:teuthology.orchestra.run.smithi057.stdout: bluestore allocator = hybrid 2023-12-02T02:47:12.330 INFO:teuthology.orchestra.run.smithi057.stdout: bluestore block size = 96636764160 2023-12-02T02:47:12.330 INFO:teuthology.orchestra.run.smithi057.stdout: bluestore fsck on mount = True 2023-12-02T02:47:12.330 INFO:teuthology.orchestra.run.smithi057.stdout: debug bluefs = 1/20 2023-12-02T02:47:12.330 INFO:teuthology.orchestra.run.smithi057.stdout: debug bluestore = 1/20 2023-12-02T02:47:12.330 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2023-12-02T02:47:12.330 INFO:teuthology.orchestra.run.smithi057.stdout: debug osd = 20 2023-12-02T02:47:12.330 INFO:teuthology.orchestra.run.smithi057.stdout: debug rocksdb = 4/10 2023-12-02T02:47:12.330 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd backfillfull_ratio = 0.85 2023-12-02T02:47:12.330 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd full ratio = 0.9 2023-12-02T02:47:12.331 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd nearfull ratio = 0.8 2023-12-02T02:47:12.331 INFO:teuthology.orchestra.run.smithi057.stdout: osd failsafe full ratio = 0.95 2023-12-02T02:47:12.331 INFO:teuthology.orchestra.run.smithi057.stdout: osd objectstore = bluestore 2023-12-02T02:47:12.331 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.331 INFO:teuthology.orchestra.run.smithi057.stdout:[mgr] 2023-12-02T02:47:12.331 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2023-12-02T02:47:12.331 INFO:teuthology.orchestra.run.smithi057.stdout: debug mgr = 20 2023-12-02T02:47:12.331 INFO:teuthology.orchestra.run.smithi057.stdout: debug mon = 20 2023-12-02T02:47:12.331 INFO:teuthology.orchestra.run.smithi057.stdout: debug auth = 20 2023-12-02T02:47:12.331 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min pgs per osd = 4 2023-12-02T02:47:12.331 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min bytes per osd = 10 2023-12-02T02:47:12.331 INFO:teuthology.orchestra.run.smithi057.stdout: mgr/telemetry/nag = false 2023-12-02T02:47:12.331 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.331 INFO:teuthology.orchestra.run.smithi057.stdout:[mon] 2023-12-02T02:47:12.331 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2023-12-02T02:47:12.331 INFO:teuthology.orchestra.run.smithi057.stdout: debug mon = 20 2023-12-02T02:47:12.332 INFO:teuthology.orchestra.run.smithi057.stdout: debug paxos = 20 2023-12-02T02:47:12.332 INFO:teuthology.orchestra.run.smithi057.stdout: debug auth = 20 2023-12-02T02:47:12.332 INFO:teuthology.orchestra.run.smithi057.stdout: mon data avail warn = 5 2023-12-02T02:47:12.332 INFO:teuthology.orchestra.run.smithi057.stdout: mon mgr mkfs grace = 240 2023-12-02T02:47:12.332 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min pgs per osd = 4 2023-12-02T02:47:12.332 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd reporter subtree level = osd 2023-12-02T02:47:12.332 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd prime pg temp = true 2023-12-02T02:47:12.332 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min bytes per osd = 10 2023-12-02T02:47:12.332 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.332 INFO:teuthology.orchestra.run.smithi057.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-02T02:47:12.332 INFO:teuthology.orchestra.run.smithi057.stdout: auth mon ticket ttl = 660 # 11m 2023-12-02T02:47:12.332 INFO:teuthology.orchestra.run.smithi057.stdout: auth service ticket ttl = 240 # 4m 2023-12-02T02:47:12.332 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.332 INFO:teuthology.orchestra.run.smithi057.stdout: # don't complain about insecure global_id in the test suite 2023-12-02T02:47:12.332 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-02T02:47:12.332 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-02T02:47:12.333 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.333 INFO:teuthology.orchestra.run.smithi057.stdout: # 1m isn't quite enough 2023-12-02T02:47:12.333 INFO:teuthology.orchestra.run.smithi057.stdout: mon_down_mkfs_grace = 2m 2023-12-02T02:47:12.333 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.333 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_filestore_osds = false 2023-12-02T02:47:12.333 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:12.333 INFO:teuthology.orchestra.run.smithi057.stdout:[client] 2023-12-02T02:47:12.333 INFO:teuthology.orchestra.run.smithi057.stdout: rgw cache enabled = true 2023-12-02T02:47:12.333 INFO:teuthology.orchestra.run.smithi057.stdout: rgw enable ops log = true 2023-12-02T02:47:12.333 INFO:teuthology.orchestra.run.smithi057.stdout: rgw enable usage log = true 2023-12-02T02:47:12.333 INFO:teuthology.orchestra.run.smithi057.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-02T02:47:12.333 INFO:teuthology.orchestra.run.smithi057.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-02T02:47:12.333 INFO:teuthology.orchestra.run.smithi057.stdout: debug rbd = 20 2023-12-02T02:47:12.333 INFO:teuthology.orchestra.run.smithi057.stdout: debug rbd_mirror = 15 2023-12-02T02:47:12.333 INFO:teuthology.orchestra.run.smithi057.stdout: log to stderr = False 2023-12-02T02:47:12.334 INFO:teuthology.orchestra.run.smithi057.stdout: rbd default features = 125 2023-12-02T02:47:12.334 INFO:teuthology.orchestra.run.smithi057.stdout: rbd mirror image policy type = simple 2023-12-02T02:47:12.334 INFO:teuthology.orchestra.run.smithi057.stdout:[mon.a] 2023-12-02T02:47:12.334 INFO:teuthology.orchestra.run.smithi057.stdout:[client.mirror.0] 2023-12-02T02:47:12.334 INFO:teuthology.orchestra.run.smithi057.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T02:47:12.334 INFO:teuthology.orchestra.run.smithi057.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T02:47:12.334 INFO:teuthology.orchestra.run.smithi057.stdout:[client.mirror.1] 2023-12-02T02:47:12.334 INFO:teuthology.orchestra.run.smithi057.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T02:47:12.334 INFO:teuthology.orchestra.run.smithi057.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T02:47:12.334 INFO:teuthology.orchestra.run.smithi057.stdout:[client.mirror.2] 2023-12-02T02:47:12.335 INFO:teuthology.orchestra.run.smithi057.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T02:47:12.335 INFO:teuthology.orchestra.run.smithi057.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T02:47:12.335 INFO:teuthology.orchestra.run.smithi057.stdout:[client.mirror.3] 2023-12-02T02:47:12.335 INFO:teuthology.orchestra.run.smithi057.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T02:47:12.335 INFO:teuthology.orchestra.run.smithi057.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T02:47:12.336 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2023-12-02T02:47:12.336 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2023-12-02T02:47:12.435 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-02T02:47:12.436 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-02T02:47:12.436 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2023-12-02T02:47:12.451 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-02T02:47:12.451 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2023-12-02T02:47:12.506 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi029.front.sepia.ceph.com 2023-12-02T02:47:12.506 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-02T02:47:12.506 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/cluster1.keyring 2023-12-02T02:47:12.506 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-02T02:47:12.593 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-02T02:47:12.593 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-12-02T02:47:12.648 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi057.front.sepia.ceph.com 2023-12-02T02:47:12.649 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:47:12.649 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/ceph/cluster1.keyring 2023-12-02T02:47:12.649 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-02T02:47:12.697 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:47:12.697 DEBUG:teuthology.orchestra.run.smithi057:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-12-02T02:47:12.753 INFO:tasks.ceph:Setting up mon nodes... 2023-12-02T02:47:12.753 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-02T02:47:12.753 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2023-12-02T02:47:12.815 INFO:teuthology.orchestra.run.smithi029.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2023-12-02T02:47:12.818 INFO:tasks.ceph:Setting up mds nodes... 2023-12-02T02:47:12.818 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-02T02:47:12.818 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2023-12-02T02:47:12.864 INFO:teuthology.orchestra.run.smithi029.stdout:creating /etc/ceph/cluster1.client.0.keyring 2023-12-02T02:47:12.883 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2023-12-02T02:47:12.937 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2023-12-02T02:47:12.956 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2023-12-02T02:47:13.002 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2023-12-02T02:47:13.020 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2023-12-02T02:47:13.066 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2023-12-02T02:47:13.085 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2023-12-02T02:47:13.130 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2023-12-02T02:47:13.149 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2023-12-02T02:47:13.194 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2023-12-02T02:47:13.213 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2023-12-02T02:47:13.258 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2023-12-02T02:47:13.277 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2023-12-02T02:47:13.323 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2023-12-02T02:47:13.342 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2023-12-02T02:47:13.388 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2023-12-02T02:47:13.406 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-02T02:47:13.406 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi029.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-02T02:47:13.407 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2023-12-02T02:47:13.439 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-02T02:47:13.439 INFO:tasks.ceph:role: cluster1.osd.0 2023-12-02T02:47:13.440 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi029.front.sepia.ceph.com 2023-12-02T02:47:13.440 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-02T02:47:13.511 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-02T02:47:13.511 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-02T02:47:13.511 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-02T02:47:13.512 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-02T02:47:13.512 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-02T02:47:13.512 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2023-12-02T02:47:13.512 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-02T02:47:13.512 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-02T02:47:13.512 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-02T02:47:13.512 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-02T02:47:13.543 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2023-12-02T02:47:13.545 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2023-12-02T02:47:13.545 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2023-12-02T02:47:13.619 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2023-12-02T02:47:13.694 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2023-12-02T02:47:13.766 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-02T02:47:13.766 INFO:tasks.ceph:role: cluster1.osd.1 2023-12-02T02:47:13.766 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi029.front.sepia.ceph.com 2023-12-02T02:47:13.766 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-02T02:47:13.836 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-02T02:47:13.837 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-02T02:47:13.837 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-02T02:47:13.837 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-02T02:47:13.837 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-02T02:47:13.837 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2023-12-02T02:47:13.837 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-02T02:47:13.837 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-02T02:47:13.837 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-02T02:47:13.837 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-02T02:47:13.868 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2023-12-02T02:47:13.870 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2023-12-02T02:47:13.870 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2023-12-02T02:47:13.945 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2023-12-02T02:47:14.020 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2023-12-02T02:47:14.091 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-02T02:47:14.091 INFO:tasks.ceph:role: cluster1.osd.2 2023-12-02T02:47:14.091 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi029.front.sepia.ceph.com 2023-12-02T02:47:14.091 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-02T02:47:14.161 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-02T02:47:14.162 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-02T02:47:14.162 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-02T02:47:14.162 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-02T02:47:14.162 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-02T02:47:14.162 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2023-12-02T02:47:14.162 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-02T02:47:14.162 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-02T02:47:14.162 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-02T02:47:14.162 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-02T02:47:14.194 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2023-12-02T02:47:14.196 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2023-12-02T02:47:14.196 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2023-12-02T02:47:14.271 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2023-12-02T02:47:14.346 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-02T02:47:14.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-02T02:47:14.448+0000 7f22062a5500 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2023-12-02T02:47:14.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-02T02:47:14.448+0000 7f22062a5500 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2023-12-02T02:47:14.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-02T02:47:14.448+0000 7f22062a5500 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-02T02:47:14.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-02T02:47:14.448+0000 7f22062a5500 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-02T02:47:14.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-02T02:47:14.448+0000 7f22062a5500 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-02T02:47:14.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-02T02:47:14.448+0000 7f22062a5500 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2023-12-02T02:47:17.301 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2023-12-02T02:47:17.337 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-02T02:47:17.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-02T02:47:17.429+0000 7feba3c10500 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2023-12-02T02:47:17.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-02T02:47:17.429+0000 7feba3c10500 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2023-12-02T02:47:17.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-02T02:47:17.429+0000 7feba3c10500 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-02T02:47:17.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-02T02:47:17.429+0000 7feba3c10500 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-02T02:47:17.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-02T02:47:17.429+0000 7feba3c10500 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-02T02:47:17.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-02T02:47:17.429+0000 7feba3c10500 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2023-12-02T02:47:20.296 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2023-12-02T02:47:20.332 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-02T02:47:20.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-02T02:47:20.425+0000 7eff97b25500 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2023-12-02T02:47:20.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-02T02:47:20.425+0000 7eff97b25500 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2023-12-02T02:47:20.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-02T02:47:20.425+0000 7eff97b25500 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-02T02:47:20.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-02T02:47:20.425+0000 7eff97b25500 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-02T02:47:20.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-02T02:47:20.425+0000 7eff97b25500 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-02T02:47:20.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-12-02T02:47:20.425+0000 7eff97b25500 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2023-12-02T02:47:23.286 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2023-12-02T02:47:23.324 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-02T02:47:23.324 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-02T02:47:23.324 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2023-12-02T02:47:23.398 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-02T02:47:23.398 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2023-12-02T02:47:23.471 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-02T02:47:23.471 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2023-12-02T02:47:23.543 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-02T02:47:23.543 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2023-12-02T02:47:23.614 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-02T02:47:23.614 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2023-12-02T02:47:23.670 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:47:23.670 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2023-12-02T02:47:23.689 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:47:23.689 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2023-12-02T02:47:23.748 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:47:23.748 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2023-12-02T02:47:23.806 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:47:23.806 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2023-12-02T02:47:23.864 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:47:23.864 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2023-12-02T02:47:23.921 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:47:23.922 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2023-12-02T02:47:23.979 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:47:23.979 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2023-12-02T02:47:24.037 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:47:24.037 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2023-12-02T02:47:24.094 INFO:tasks.ceph:Adding keys to all mons... 2023-12-02T02:47:24.095 DEBUG:teuthology.orchestra.run.smithi029:> sudo tee -a /etc/ceph/cluster1.keyring 2023-12-02T02:47:24.126 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr.x] 2023-12-02T02:47:24.126 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQCwmmpllt+DMBAAw0KlRpCWYQ8M6Hpe/9c1xQ== 2023-12-02T02:47:24.126 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.0] 2023-12-02T02:47:24.127 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQCymmplmC/IGhAAEvUZTV+4VPk1psQCHWCi0Q== 2023-12-02T02:47:24.127 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.1] 2023-12-02T02:47:24.127 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQC1mmplhzmnGRAAmwjEBFqXMHYoBjO8h6ZfkA== 2023-12-02T02:47:24.127 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.2] 2023-12-02T02:47:24.127 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQC4mmplH0ZsGRAAycev3OLQNHcE8d+7L06pxA== 2023-12-02T02:47:24.127 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-12-02T02:47:24.127 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQCwmmplQadsMxAALACkROKLykYPV88xn4bhyw== 2023-12-02T02:47:24.127 INFO:teuthology.orchestra.run.smithi029.stdout:[client.mirror] 2023-12-02T02:47:24.127 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQCwmmpltZfGNxAAUDELwC8t5wEVpF6awg2ziw== 2023-12-02T02:47:24.127 INFO:teuthology.orchestra.run.smithi029.stdout:[client.mirror.0] 2023-12-02T02:47:24.127 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQCxmmpldJwCABAAaY+tg5WkEoZozXkdq6PvVA== 2023-12-02T02:47:24.127 INFO:teuthology.orchestra.run.smithi029.stdout:[client.mirror.1] 2023-12-02T02:47:24.127 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQCxmmpl/KzZAxAAAHNU5BxNOS4Hb7mlP8xfiQ== 2023-12-02T02:47:24.128 INFO:teuthology.orchestra.run.smithi029.stdout:[client.mirror.2] 2023-12-02T02:47:24.128 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQCxmmplYLqpBxAAyX536ockZP9zBJos7q7wtA== 2023-12-02T02:47:24.128 INFO:teuthology.orchestra.run.smithi029.stdout:[client.mirror.3] 2023-12-02T02:47:24.128 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQCxmmplDH9+CxAA7e5cNOlSchPQ2fLTpCCDQA== 2023-12-02T02:47:24.128 INFO:teuthology.orchestra.run.smithi029.stdout:[client.mirror.4] 2023-12-02T02:47:24.128 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQCxmmpl0f5ODxAAXEIIZ+xPhScTMbBetD5/+w== 2023-12-02T02:47:24.128 INFO:teuthology.orchestra.run.smithi029.stdout:[client.mirror.5] 2023-12-02T02:47:24.128 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQCxmmplqz8vExAAF5XPGfAIgqgBoZf6f/qotw== 2023-12-02T02:47:24.128 INFO:teuthology.orchestra.run.smithi029.stdout:[client.mirror.6] 2023-12-02T02:47:24.128 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQCxmmpl8koIFxAAPwQrvQ20cagI+xEDL0STvA== 2023-12-02T02:47:24.129 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-02T02:47:24.228 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-02T02:47:24.285 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-02T02:47:24.341 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-02T02:47:24.397 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T02:47:24.455 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T02:47:24.515 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T02:47:24.574 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T02:47:24.633 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T02:47:24.693 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T02:47:24.753 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T02:47:24.812 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T02:47:24.871 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T02:47:24.930 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-02T02:47:24.930 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2023-12-02T02:47:24.960 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2023-12-02T02:47:25.062 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2023-12-02T02:47:25.092 DEBUG:teuthology.orchestra.run.smithi029:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2023-12-02T02:47:25.147 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2023-12-02T02:47:25.147 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2023-12-02T02:47:25.147 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2023-12-02T02:47:25.189 INFO:tasks.ceph.cluster1.mon.a:Started 2023-12-02T02:47:25.190 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2023-12-02T02:47:25.190 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2023-12-02T02:47:25.190 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2023-12-02T02:47:25.192 INFO:tasks.ceph.cluster1.mgr.x:Started 2023-12-02T02:47:25.192 INFO:tasks.ceph:Setting crush tunables to default 2023-12-02T02:47:25.192 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster cluster1 osd crush tunables default 2023-12-02T02:47:25.384 INFO:teuthology.orchestra.run.smithi029.stderr:adjusted tunables profile to default 2023-12-02T02:47:25.395 INFO:tasks.ceph:check_enable_crimson: False 2023-12-02T02:47:25.396 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2023-12-02T02:47:25.396 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-02T02:47:25.396 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2023-12-02T02:47:25.427 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-02T02:47:25.427 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2023-12-02T02:47:25.497 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-02T02:47:25.498 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2023-12-02T02:47:25.569 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster cluster1 osd new e6692581-4f3d-46c3-9ad5-096568abc978 0 2023-12-02T02:47:25.796 INFO:teuthology.orchestra.run.smithi029.stdout:0 2023-12-02T02:47:25.807 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster cluster1 osd new 22a5db65-fbac-4ba3-aaf6-b9a1560a8c3b 1 2023-12-02T02:47:25.955 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T02:47:25.952+0000 7f03ebe32040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-02T02:47:25.996 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-12-02T02:47:26.007 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster cluster1 osd new 5ca69845-ddc0-4fa1-b94d-b36a3b49a7d4 2 2023-12-02T02:47:26.010 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T02:47:26.007+0000 7f03ebe32040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-02T02:47:26.208 INFO:teuthology.orchestra.run.smithi029.stdout:2 2023-12-02T02:47:26.220 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2023-12-02T02:47:26.220 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2023-12-02T02:47:26.263 INFO:tasks.ceph.cluster1.osd.0:Started 2023-12-02T02:47:26.263 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2023-12-02T02:47:26.263 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2023-12-02T02:47:26.266 INFO:tasks.ceph.cluster1.osd.1:Started 2023-12-02T02:47:26.266 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2023-12-02T02:47:26.267 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2023-12-02T02:47:26.269 INFO:tasks.ceph.cluster1.osd.2:Started 2023-12-02T02:47:26.270 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-02T02:47:26.453 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:26.454 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":5,"fsid":"69e9688d-4570-4fa3-a52d-d6c0eaea7d57","created":"2023-12-02T02:47:25.302301+0000","modified":"2023-12-02T02:47:26.206215+0000","last_up_change":"0.000000","last_in_change":"2023-12-02T02:47:26.206215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e6692581-4f3d-46c3-9ad5-096568abc978","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"22a5db65-fbac-4ba3-aaf6-b9a1560a8c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ca69845-ddc0-4fa1-b94d-b36a3b49a7d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-02T02:47:26.465 INFO:tasks.ceph.ceph_manager.cluster1:[] 2023-12-02T02:47:26.465 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-02T02:47:26.632 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T02:47:26.629+0000 7f03ebe32040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-02T02:47:26.687 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T02:47:26.684+0000 7f03ebe32040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-02T02:47:26.734 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T02:47:26.730+0000 7f03ebe32040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-02T02:47:26.767 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-12-02T02:47:26.791 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T02:47:26.788+0000 7f03ebe32040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-02T02:47:26.836 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T02:47:26.833+0000 7f03ebe32040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-02T02:47:26.865 INFO:tasks.ceph.cluster1.osd.1.smithi029.stderr:2023-12-02T02:47:26.861+0000 7f5a78ca3500 -1 Falling back to public interface 2023-12-02T02:47:26.878 INFO:tasks.ceph.cluster1.osd.2.smithi029.stderr:2023-12-02T02:47:26.874+0000 7f13c680e500 -1 Falling back to public interface 2023-12-02T02:47:26.896 INFO:tasks.ceph.cluster1.osd.0.smithi029.stderr:2023-12-02T02:47:26.893+0000 7f5f3bed2500 -1 Falling back to public interface 2023-12-02T02:47:26.931 INFO:teuthology.misc.health.smithi029.stdout: 2023-12-02T02:47:26.931 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":5,"fsid":"69e9688d-4570-4fa3-a52d-d6c0eaea7d57","created":"2023-12-02T02:47:25.302301+0000","modified":"2023-12-02T02:47:26.206215+0000","last_up_change":"0.000000","last_in_change":"2023-12-02T02:47:26.206215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e6692581-4f3d-46c3-9ad5-096568abc978","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"22a5db65-fbac-4ba3-aaf6-b9a1560a8c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ca69845-ddc0-4fa1-b94d-b36a3b49a7d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-02T02:47:26.941 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-02T02:47:26.951 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T02:47:26.948+0000 7f03ebe32040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-02T02:47:27.024 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T02:47:27.021+0000 7f03ebe32040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-02T02:47:27.072 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T02:47:27.068+0000 7f03ebe32040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-02T02:47:27.126 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T02:47:27.122+0000 7f03ebe32040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-02T02:47:27.175 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T02:47:27.172+0000 7f03ebe32040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-02T02:47:27.575 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T02:47:27.571+0000 7f03ebe32040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-02T02:47:27.717 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T02:47:27.713+0000 7f03ebe32040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-02T02:47:27.838 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T02:47:27.835+0000 7f03ebe32040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-02T02:47:27.885 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T02:47:27.882+0000 7f03ebe32040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-02T02:47:27.987 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T02:47:27.984+0000 7f03ebe32040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-02T02:47:28.086 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T02:47:28.083+0000 7f03ebe32040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-02T02:47:28.232 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T02:47:28.229+0000 7f03ebe32040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-02T02:47:28.276 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T02:47:28.273+0000 7f03ebe32040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-02T02:47:28.424 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T02:47:28.420+0000 7f03ebe32040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-02T02:47:29.291 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T02:47:29.288+0000 7f03ebe32040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-02T02:47:29.583 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T02:47:29.580+0000 7f03ebe32040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-02T02:47:29.661 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T02:47:29.657+0000 7f03ebe32040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-02T02:47:29.791 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T02:47:29.787+0000 7f03ebe32040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-02T02:47:30.379 INFO:tasks.ceph.cluster1.osd.1.smithi029.stderr:2023-12-02T02:47:30.375+0000 7f5a78ca3500 -1 osd.1 0 log_to_monitors true 2023-12-02T02:47:30.381 INFO:tasks.ceph.cluster1.osd.2.smithi029.stderr:2023-12-02T02:47:30.378+0000 7f13c680e500 -1 osd.2 0 log_to_monitors true 2023-12-02T02:47:30.405 INFO:tasks.ceph.cluster1.osd.0.smithi029.stderr:2023-12-02T02:47:30.401+0000 7f5f3bed2500 -1 osd.0 0 log_to_monitors true 2023-12-02T02:47:30.810 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T02:47:30.807+0000 7f038cad7640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-02T02:47:33.245 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-12-02T02:47:33.597 INFO:tasks.ceph.cluster1.osd.0.smithi029.stderr:2023-12-02T02:47:33.594+0000 7f5f37e60640 -1 osd.0 0 waiting for initial osdmap 2023-12-02T02:47:33.612 INFO:tasks.ceph.cluster1.osd.0.smithi029.stderr:2023-12-02T02:47:33.609+0000 7f5f2fc86640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-02T02:47:33.666 INFO:tasks.ceph.cluster1.osd.1.smithi029.stderr:2023-12-02T02:47:33.663+0000 7f5a75445640 -1 osd.1 0 waiting for initial osdmap 2023-12-02T02:47:33.683 INFO:tasks.ceph.cluster1.osd.2.smithi029.stderr:2023-12-02T02:47:33.680+0000 7f13c279c640 -1 osd.2 0 waiting for initial osdmap 2023-12-02T02:47:33.686 INFO:tasks.ceph.cluster1.osd.1.smithi029.stderr:2023-12-02T02:47:33.683+0000 7f5a6ca57640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-02T02:47:33.701 INFO:tasks.ceph.cluster1.osd.2.smithi029.stderr:2023-12-02T02:47:33.698+0000 7f13ba5c2640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-02T02:47:33.724 INFO:teuthology.misc.health.smithi029.stdout: 2023-12-02T02:47:33.725 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":7,"fsid":"69e9688d-4570-4fa3-a52d-d6c0eaea7d57","created":"2023-12-02T02:47:25.302301+0000","modified":"2023-12-02T02:47:32.278697+0000","last_up_change":"0.000000","last_in_change":"2023-12-02T02:47:26.206215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e6692581-4f3d-46c3-9ad5-096568abc978","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"22a5db65-fbac-4ba3-aaf6-b9a1560a8c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ca69845-ddc0-4fa1-b94d-b36a3b49a7d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-02T02:47:33.736 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-02T02:47:40.040 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-12-02T02:47:40.337 INFO:teuthology.misc.health.smithi029.stdout: 2023-12-02T02:47:40.337 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":11,"fsid":"69e9688d-4570-4fa3-a52d-d6c0eaea7d57","created":"2023-12-02T02:47:25.302301+0000","modified":"2023-12-02T02:47:38.295735+0000","last_up_change":"2023-12-02T02:47:34.596797+0000","last_in_change":"2023-12-02T02:47:26.206215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-02T02:47:35.863550+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6692581-4f3d-46c3-9ad5-096568abc978","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3463167137},{"type":"v1","addr":"172.21.15.29:6817","nonce":3463167137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3463167137},{"type":"v1","addr":"172.21.15.29:6819","nonce":3463167137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":3463167137},{"type":"v1","addr":"172.21.15.29:6823","nonce":3463167137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":3463167137},{"type":"v1","addr":"172.21.15.29:6821","nonce":3463167137}]},"public_addr":"172.21.15.29:6817/3463167137","cluster_addr":"172.21.15.29:6819/3463167137","heartbeat_back_addr":"172.21.15.29:6823/3463167137","heartbeat_front_addr":"172.21.15.29:6821/3463167137","state":["exists","up"]},{"osd":1,"uuid":"22a5db65-fbac-4ba3-aaf6-b9a1560a8c3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":2864054316},{"type":"v1","addr":"172.21.15.29:6801","nonce":2864054316}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2864054316},{"type":"v1","addr":"172.21.15.29:6803","nonce":2864054316}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2864054316},{"type":"v1","addr":"172.21.15.29:6807","nonce":2864054316}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2864054316},{"type":"v1","addr":"172.21.15.29:6805","nonce":2864054316}]},"public_addr":"172.21.15.29:6801/2864054316","cluster_addr":"172.21.15.29:6803/2864054316","heartbeat_back_addr":"172.21.15.29:6807/2864054316","heartbeat_front_addr":"172.21.15.29:6805/2864054316","state":["exists","up"]},{"osd":2,"uuid":"5ca69845-ddc0-4fa1-b94d-b36a3b49a7d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3794819590},{"type":"v1","addr":"172.21.15.29:6809","nonce":3794819590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3794819590},{"type":"v1","addr":"172.21.15.29:6811","nonce":3794819590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3794819590},{"type":"v1","addr":"172.21.15.29:6815","nonce":3794819590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3794819590},{"type":"v1","addr":"172.21.15.29:6813","nonce":3794819590}]},"public_addr":"172.21.15.29:6809/3794819590","cluster_addr":"172.21.15.29:6811/3794819590","heartbeat_back_addr":"172.21.15.29:6815/3794819590","heartbeat_front_addr":"172.21.15.29:6813/3794819590","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T02:47:31.453138+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T02:47:31.414738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T02:47:31.343565+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-02T02:47:40.349 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-12-02T02:47:40.349 INFO:tasks.ceph:Creating RBD pool 2023-12-02T02:47:40.349 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster cluster1 osd pool create rbd 8 2023-12-02T02:47:41.304 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'rbd' created 2023-12-02T02:47:41.322 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-02T02:47:42.307 INFO:teuthology.orchestra.run.smithi029.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-02T02:47:42.327 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2023-12-02T02:47:42.327 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2023-12-02T02:47:42.328 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2023-12-02T02:47:42.328 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2023-12-02T02:47:42.328 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-02T02:47:42.675 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:42.687 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":5,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3869149468},{"type":"v1","addr":"172.21.15.29:6825","nonce":3869149468}]},"active_addr":"172.21.15.29:6825/3869149468","active_change":"2023-12-02T02:47:29.801158+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":3260877794}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":3581000734}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":1902324912}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":913233963}]}]} 2023-12-02T02:47:42.689 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2023-12-02T02:47:42.689 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2023-12-02T02:47:42.689 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-02T02:47:42.983 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:42.983 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":13,"fsid":"69e9688d-4570-4fa3-a52d-d6c0eaea7d57","created":"2023-12-02T02:47:25.302301+0000","modified":"2023-12-02T02:47:42.302678+0000","last_up_change":"2023-12-02T02:47:34.596797+0000","last_in_change":"2023-12-02T02:47:26.206215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-02T02:47:35.863550+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-02T02:47:40.672535+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6692581-4f3d-46c3-9ad5-096568abc978","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3463167137},{"type":"v1","addr":"172.21.15.29:6817","nonce":3463167137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3463167137},{"type":"v1","addr":"172.21.15.29:6819","nonce":3463167137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":3463167137},{"type":"v1","addr":"172.21.15.29:6823","nonce":3463167137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":3463167137},{"type":"v1","addr":"172.21.15.29:6821","nonce":3463167137}]},"public_addr":"172.21.15.29:6817/3463167137","cluster_addr":"172.21.15.29:6819/3463167137","heartbeat_back_addr":"172.21.15.29:6823/3463167137","heartbeat_front_addr":"172.21.15.29:6821/3463167137","state":["exists","up"]},{"osd":1,"uuid":"22a5db65-fbac-4ba3-aaf6-b9a1560a8c3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":2864054316},{"type":"v1","addr":"172.21.15.29:6801","nonce":2864054316}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2864054316},{"type":"v1","addr":"172.21.15.29:6803","nonce":2864054316}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2864054316},{"type":"v1","addr":"172.21.15.29:6807","nonce":2864054316}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2864054316},{"type":"v1","addr":"172.21.15.29:6805","nonce":2864054316}]},"public_addr":"172.21.15.29:6801/2864054316","cluster_addr":"172.21.15.29:6803/2864054316","heartbeat_back_addr":"172.21.15.29:6807/2864054316","heartbeat_front_addr":"172.21.15.29:6805/2864054316","state":["exists","up"]},{"osd":2,"uuid":"5ca69845-ddc0-4fa1-b94d-b36a3b49a7d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3794819590},{"type":"v1","addr":"172.21.15.29:6809","nonce":3794819590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3794819590},{"type":"v1","addr":"172.21.15.29:6811","nonce":3794819590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3794819590},{"type":"v1","addr":"172.21.15.29:6815","nonce":3794819590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3794819590},{"type":"v1","addr":"172.21.15.29:6813","nonce":3794819590}]},"public_addr":"172.21.15.29:6809/3794819590","cluster_addr":"172.21.15.29:6811/3794819590","heartbeat_back_addr":"172.21.15.29:6815/3794819590","heartbeat_front_addr":"172.21.15.29:6813/3794819590","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T02:47:31.453138+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T02:47:31.414738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T02:47:31.343565+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-02T02:47:42.995 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2023-12-02T02:47:42.996 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-02T02:47:43.290 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:43.291 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":13,"fsid":"69e9688d-4570-4fa3-a52d-d6c0eaea7d57","created":"2023-12-02T02:47:25.302301+0000","modified":"2023-12-02T02:47:42.302678+0000","last_up_change":"2023-12-02T02:47:34.596797+0000","last_in_change":"2023-12-02T02:47:26.206215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-02T02:47:35.863550+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-02T02:47:40.672535+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6692581-4f3d-46c3-9ad5-096568abc978","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3463167137},{"type":"v1","addr":"172.21.15.29:6817","nonce":3463167137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3463167137},{"type":"v1","addr":"172.21.15.29:6819","nonce":3463167137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":3463167137},{"type":"v1","addr":"172.21.15.29:6823","nonce":3463167137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":3463167137},{"type":"v1","addr":"172.21.15.29:6821","nonce":3463167137}]},"public_addr":"172.21.15.29:6817/3463167137","cluster_addr":"172.21.15.29:6819/3463167137","heartbeat_back_addr":"172.21.15.29:6823/3463167137","heartbeat_front_addr":"172.21.15.29:6821/3463167137","state":["exists","up"]},{"osd":1,"uuid":"22a5db65-fbac-4ba3-aaf6-b9a1560a8c3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":2864054316},{"type":"v1","addr":"172.21.15.29:6801","nonce":2864054316}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2864054316},{"type":"v1","addr":"172.21.15.29:6803","nonce":2864054316}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2864054316},{"type":"v1","addr":"172.21.15.29:6807","nonce":2864054316}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2864054316},{"type":"v1","addr":"172.21.15.29:6805","nonce":2864054316}]},"public_addr":"172.21.15.29:6801/2864054316","cluster_addr":"172.21.15.29:6803/2864054316","heartbeat_back_addr":"172.21.15.29:6807/2864054316","heartbeat_front_addr":"172.21.15.29:6805/2864054316","state":["exists","up"]},{"osd":2,"uuid":"5ca69845-ddc0-4fa1-b94d-b36a3b49a7d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3794819590},{"type":"v1","addr":"172.21.15.29:6809","nonce":3794819590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3794819590},{"type":"v1","addr":"172.21.15.29:6811","nonce":3794819590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3794819590},{"type":"v1","addr":"172.21.15.29:6815","nonce":3794819590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3794819590},{"type":"v1","addr":"172.21.15.29:6813","nonce":3794819590}]},"public_addr":"172.21.15.29:6809/3794819590","cluster_addr":"172.21.15.29:6811/3794819590","heartbeat_back_addr":"172.21.15.29:6815/3794819590","heartbeat_front_addr":"172.21.15.29:6813/3794819590","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T02:47:31.453138+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T02:47:31.414738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T02:47:31.343565+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-02T02:47:43.305 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2023-12-02T02:47:43.305 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2023-12-02T02:47:43.306 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2023-12-02T02:47:43.454 INFO:teuthology.orchestra.run.smithi029.stdout:34359738373 2023-12-02T02:47:43.455 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-12-02T02:47:43.474 INFO:teuthology.orchestra.run.smithi029.stdout:34359738372 2023-12-02T02:47:43.474 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-12-02T02:47:43.475 INFO:teuthology.orchestra.run.smithi029.stdout:34359738372 2023-12-02T02:47:43.475 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-12-02T02:47:43.764 INFO:teuthology.orchestra.run.smithi029.stdout:34359738372 2023-12-02T02:47:43.777 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.1 2023-12-02T02:47:43.837 INFO:teuthology.orchestra.run.smithi029.stdout:34359738371 2023-12-02T02:47:43.840 INFO:teuthology.orchestra.run.smithi029.stdout:34359738371 2023-12-02T02:47:43.850 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.0 2023-12-02T02:47:43.853 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.2 2023-12-02T02:47:44.777 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-12-02T02:47:44.851 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-12-02T02:47:44.853 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-12-02T02:47:45.094 INFO:teuthology.orchestra.run.smithi029.stdout:34359738373 2023-12-02T02:47:45.107 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.1 2023-12-02T02:47:45.107 DEBUG:teuthology.parallel:result is None 2023-12-02T02:47:45.161 INFO:teuthology.orchestra.run.smithi029.stdout:34359738372 2023-12-02T02:47:45.165 INFO:teuthology.orchestra.run.smithi029.stdout:34359738372 2023-12-02T02:47:45.174 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738372 for osd.2 2023-12-02T02:47:45.174 DEBUG:teuthology.parallel:result is None 2023-12-02T02:47:45.176 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738372 for osd.0 2023-12-02T02:47:45.176 DEBUG:teuthology.parallel:result is None 2023-12-02T02:47:45.176 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2023-12-02T02:47:45.177 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-02T02:47:45.506 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:45.507 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-12-02T02:47:45.518 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-12-02T02:47:43.807010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":95040,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283020480,"statfs":{"total":289910292480,"available":289812971520,"internally_reserved":0,"allocated":1531904,"data_stored":1274729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.504337"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-02T02:47:42.315316+0000","last_change":"2023-12-02T02:47:42.314048+0000","last_active":"2023-12-02T02:47:42.315316+0000","last_peered":"2023-12-02T02:47:42.315316+0000","last_clean":"2023-12-02T02:47:42.315316+0000","last_became_active":"2023-12-02T02:47:42.313408+0000","last_became_peered":"2023-12-02T02:47:42.313408+0000","last_unstale":"2023-12-02T02:47:42.315316+0000","last_undegraded":"2023-12-02T02:47:42.315316+0000","last_fullsized":"2023-12-02T02:47:42.315316+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_clean_scrub_stamp":"2023-12-02T02:47:41.299292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T05:40:28.917735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-02T02:47:42.315274+0000","last_change":"2023-12-02T02:47:42.313619+0000","last_active":"2023-12-02T02:47:42.315274+0000","last_peered":"2023-12-02T02:47:42.315274+0000","last_clean":"2023-12-02T02:47:42.315274+0000","last_became_active":"2023-12-02T02:47:42.312942+0000","last_became_peered":"2023-12-02T02:47:42.312942+0000","last_unstale":"2023-12-02T02:47:42.315274+0000","last_undegraded":"2023-12-02T02:47:42.315274+0000","last_fullsized":"2023-12-02T02:47:42.315274+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_clean_scrub_stamp":"2023-12-02T02:47:41.299292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T03:27:31.800066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-02T02:47:42.314587+0000","last_change":"2023-12-02T02:47:42.312507+0000","last_active":"2023-12-02T02:47:42.314587+0000","last_peered":"2023-12-02T02:47:42.314587+0000","last_clean":"2023-12-02T02:47:42.314587+0000","last_became_active":"2023-12-02T02:47:42.312233+0000","last_became_peered":"2023-12-02T02:47:42.312233+0000","last_unstale":"2023-12-02T02:47:42.314587+0000","last_undegraded":"2023-12-02T02:47:42.314587+0000","last_fullsized":"2023-12-02T02:47:42.314587+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_clean_scrub_stamp":"2023-12-02T02:47:41.299292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T11:56:20.988019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-02T02:47:42.315306+0000","last_change":"2023-12-02T02:47:42.314033+0000","last_active":"2023-12-02T02:47:42.315306+0000","last_peered":"2023-12-02T02:47:42.315306+0000","last_clean":"2023-12-02T02:47:42.315306+0000","last_became_active":"2023-12-02T02:47:42.312588+0000","last_became_peered":"2023-12-02T02:47:42.312588+0000","last_unstale":"2023-12-02T02:47:42.315306+0000","last_undegraded":"2023-12-02T02:47:42.315306+0000","last_fullsized":"2023-12-02T02:47:42.315306+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_clean_scrub_stamp":"2023-12-02T02:47:41.299292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T07:39:36.722802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-02T02:47:42.312600+0000","last_change":"2023-12-02T02:47:42.312065+0000","last_active":"2023-12-02T02:47:42.312600+0000","last_peered":"2023-12-02T02:47:42.312600+0000","last_clean":"2023-12-02T02:47:42.312600+0000","last_became_active":"2023-12-02T02:47:42.311767+0000","last_became_peered":"2023-12-02T02:47:42.311767+0000","last_unstale":"2023-12-02T02:47:42.312600+0000","last_undegraded":"2023-12-02T02:47:42.312600+0000","last_fullsized":"2023-12-02T02:47:42.312600+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_clean_scrub_stamp":"2023-12-02T02:47:41.299292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T07:33:51.231073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T02:47:43.306788+0000","last_change":"2023-12-02T02:47:42.313050+0000","last_active":"2023-12-02T02:47:43.306788+0000","last_peered":"2023-12-02T02:47:43.306788+0000","last_clean":"2023-12-02T02:47:43.306788+0000","last_became_active":"2023-12-02T02:47:42.312755+0000","last_became_peered":"2023-12-02T02:47:42.312755+0000","last_unstale":"2023-12-02T02:47:43.306788+0000","last_undegraded":"2023-12-02T02:47:43.306788+0000","last_fullsized":"2023-12-02T02:47:43.306788+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_clean_scrub_stamp":"2023-12-02T02:47:41.299292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T09:59:13.111229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T02:47:43.306798+0000","last_change":"2023-12-02T02:47:42.312409+0000","last_active":"2023-12-02T02:47:43.306798+0000","last_peered":"2023-12-02T02:47:43.306798+0000","last_clean":"2023-12-02T02:47:43.306798+0000","last_became_active":"2023-12-02T02:47:42.312157+0000","last_became_peered":"2023-12-02T02:47:42.312157+0000","last_unstale":"2023-12-02T02:47:43.306798+0000","last_undegraded":"2023-12-02T02:47:43.306798+0000","last_fullsized":"2023-12-02T02:47:43.306798+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_clean_scrub_stamp":"2023-12-02T02:47:41.299292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T07:38:48.626423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-02T02:47:42.313530+0000","last_change":"2023-12-02T02:47:42.311361+0000","last_active":"2023-12-02T02:47:42.313530+0000","last_peered":"2023-12-02T02:47:42.313530+0000","last_clean":"2023-12-02T02:47:42.313530+0000","last_became_active":"2023-12-02T02:47:42.310948+0000","last_became_peered":"2023-12-02T02:47:42.310948+0000","last_unstale":"2023-12-02T02:47:42.313530+0000","last_undegraded":"2023-12-02T02:47:42.313530+0000","last_fullsized":"2023-12-02T02:47:42.313530+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_clean_scrub_stamp":"2023-12-02T02:47:41.299292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T08:30:49.148927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-02T02:47:41.304300+0000","last_change":"2023-12-02T02:47:37.330326+0000","last_active":"2023-12-02T02:47:41.304300+0000","last_peered":"2023-12-02T02:47:41.304300+0000","last_clean":"2023-12-02T02:47:41.304300+0000","last_became_active":"2023-12-02T02:47:37.329170+0000","last_became_peered":"2023-12-02T02:47:37.329170+0000","last_unstale":"2023-12-02T02:47:41.304300+0000","last_undegraded":"2023-12-02T02:47:41.304300+0000","last_fullsized":"2023-12-02T02:47:41.304300+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:47:36.289946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:47:36.289946+0000","last_clean_scrub_stamp":"2023-12-02T02:47:36.289946+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T08:13:16.589459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31296,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340544,"statfs":{"total":96636764160,"available":96604717056,"internally_reserved":0,"allocated":114688,"data_stored":31331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.249}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":708608,"data_stored":621699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":708608,"data_stored":621699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T02:47:45.519 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-02T02:47:45.809 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:45.809 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-12-02T02:47:45.821 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-12-02T02:47:43.807010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":95040,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283020480,"statfs":{"total":289910292480,"available":289812971520,"internally_reserved":0,"allocated":1531904,"data_stored":1274729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.504337"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-02T02:47:42.315316+0000","last_change":"2023-12-02T02:47:42.314048+0000","last_active":"2023-12-02T02:47:42.315316+0000","last_peered":"2023-12-02T02:47:42.315316+0000","last_clean":"2023-12-02T02:47:42.315316+0000","last_became_active":"2023-12-02T02:47:42.313408+0000","last_became_peered":"2023-12-02T02:47:42.313408+0000","last_unstale":"2023-12-02T02:47:42.315316+0000","last_undegraded":"2023-12-02T02:47:42.315316+0000","last_fullsized":"2023-12-02T02:47:42.315316+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_clean_scrub_stamp":"2023-12-02T02:47:41.299292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T05:40:28.917735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-02T02:47:42.315274+0000","last_change":"2023-12-02T02:47:42.313619+0000","last_active":"2023-12-02T02:47:42.315274+0000","last_peered":"2023-12-02T02:47:42.315274+0000","last_clean":"2023-12-02T02:47:42.315274+0000","last_became_active":"2023-12-02T02:47:42.312942+0000","last_became_peered":"2023-12-02T02:47:42.312942+0000","last_unstale":"2023-12-02T02:47:42.315274+0000","last_undegraded":"2023-12-02T02:47:42.315274+0000","last_fullsized":"2023-12-02T02:47:42.315274+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_clean_scrub_stamp":"2023-12-02T02:47:41.299292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T03:27:31.800066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-02T02:47:42.314587+0000","last_change":"2023-12-02T02:47:42.312507+0000","last_active":"2023-12-02T02:47:42.314587+0000","last_peered":"2023-12-02T02:47:42.314587+0000","last_clean":"2023-12-02T02:47:42.314587+0000","last_became_active":"2023-12-02T02:47:42.312233+0000","last_became_peered":"2023-12-02T02:47:42.312233+0000","last_unstale":"2023-12-02T02:47:42.314587+0000","last_undegraded":"2023-12-02T02:47:42.314587+0000","last_fullsized":"2023-12-02T02:47:42.314587+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_clean_scrub_stamp":"2023-12-02T02:47:41.299292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T11:56:20.988019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-02T02:47:42.315306+0000","last_change":"2023-12-02T02:47:42.314033+0000","last_active":"2023-12-02T02:47:42.315306+0000","last_peered":"2023-12-02T02:47:42.315306+0000","last_clean":"2023-12-02T02:47:42.315306+0000","last_became_active":"2023-12-02T02:47:42.312588+0000","last_became_peered":"2023-12-02T02:47:42.312588+0000","last_unstale":"2023-12-02T02:47:42.315306+0000","last_undegraded":"2023-12-02T02:47:42.315306+0000","last_fullsized":"2023-12-02T02:47:42.315306+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_clean_scrub_stamp":"2023-12-02T02:47:41.299292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T07:39:36.722802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-02T02:47:42.312600+0000","last_change":"2023-12-02T02:47:42.312065+0000","last_active":"2023-12-02T02:47:42.312600+0000","last_peered":"2023-12-02T02:47:42.312600+0000","last_clean":"2023-12-02T02:47:42.312600+0000","last_became_active":"2023-12-02T02:47:42.311767+0000","last_became_peered":"2023-12-02T02:47:42.311767+0000","last_unstale":"2023-12-02T02:47:42.312600+0000","last_undegraded":"2023-12-02T02:47:42.312600+0000","last_fullsized":"2023-12-02T02:47:42.312600+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_clean_scrub_stamp":"2023-12-02T02:47:41.299292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T07:33:51.231073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T02:47:43.306788+0000","last_change":"2023-12-02T02:47:42.313050+0000","last_active":"2023-12-02T02:47:43.306788+0000","last_peered":"2023-12-02T02:47:43.306788+0000","last_clean":"2023-12-02T02:47:43.306788+0000","last_became_active":"2023-12-02T02:47:42.312755+0000","last_became_peered":"2023-12-02T02:47:42.312755+0000","last_unstale":"2023-12-02T02:47:43.306788+0000","last_undegraded":"2023-12-02T02:47:43.306788+0000","last_fullsized":"2023-12-02T02:47:43.306788+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_clean_scrub_stamp":"2023-12-02T02:47:41.299292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T09:59:13.111229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T02:47:43.306798+0000","last_change":"2023-12-02T02:47:42.312409+0000","last_active":"2023-12-02T02:47:43.306798+0000","last_peered":"2023-12-02T02:47:43.306798+0000","last_clean":"2023-12-02T02:47:43.306798+0000","last_became_active":"2023-12-02T02:47:42.312157+0000","last_became_peered":"2023-12-02T02:47:42.312157+0000","last_unstale":"2023-12-02T02:47:43.306798+0000","last_undegraded":"2023-12-02T02:47:43.306798+0000","last_fullsized":"2023-12-02T02:47:43.306798+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_clean_scrub_stamp":"2023-12-02T02:47:41.299292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T07:38:48.626423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-02T02:47:42.313530+0000","last_change":"2023-12-02T02:47:42.311361+0000","last_active":"2023-12-02T02:47:42.313530+0000","last_peered":"2023-12-02T02:47:42.313530+0000","last_clean":"2023-12-02T02:47:42.313530+0000","last_became_active":"2023-12-02T02:47:42.310948+0000","last_became_peered":"2023-12-02T02:47:42.310948+0000","last_unstale":"2023-12-02T02:47:42.313530+0000","last_undegraded":"2023-12-02T02:47:42.313530+0000","last_fullsized":"2023-12-02T02:47:42.313530+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:47:41.299292+0000","last_clean_scrub_stamp":"2023-12-02T02:47:41.299292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T08:30:49.148927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-02T02:47:41.304300+0000","last_change":"2023-12-02T02:47:37.330326+0000","last_active":"2023-12-02T02:47:41.304300+0000","last_peered":"2023-12-02T02:47:41.304300+0000","last_clean":"2023-12-02T02:47:41.304300+0000","last_became_active":"2023-12-02T02:47:37.329170+0000","last_became_peered":"2023-12-02T02:47:37.329170+0000","last_unstale":"2023-12-02T02:47:41.304300+0000","last_undegraded":"2023-12-02T02:47:41.304300+0000","last_fullsized":"2023-12-02T02:47:41.304300+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:47:36.289946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:47:36.289946+0000","last_clean_scrub_stamp":"2023-12-02T02:47:36.289946+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T08:13:16.589459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31296,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340544,"statfs":{"total":96636764160,"available":96604717056,"internally_reserved":0,"allocated":114688,"data_stored":31331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.249}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":708608,"data_stored":621699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":708608,"data_stored":621699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T02:47:45.822 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2023-12-02T02:47:45.822 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2023-12-02T02:47:45.822 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2023-12-02T02:47:45.822 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2023-12-02T02:47:46.155 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:46.155 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-02T02:47:46.167 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2023-12-02T02:47:46.168 INFO:teuthology.run_tasks:Running task ceph... 2023-12-02T02:47:46.179 INFO:tasks.ceph:Creating ceph cluster cluster2... 2023-12-02T02:47:46.179 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-02T02:47:46.179 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-12-01_18:01:25-rbd-main-distro-default-smithi/7474803', 'branch': 'main', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7474803', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'yuriw-2023-12-01_18:01:25-rbd-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'bed7681480ccb0192b91a35e691658867e1e96b7'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'bed7681480ccb0192b91a35e691658867e1e96b7'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': 'bed7681480ccb0192b91a35e691658867e1e96b7'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 2451, 'sha1': 'bed7681480ccb0192b91a35e691658867e1e96b7', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'bed7681480ccb0192b91a35e691658867e1e96b7', 'targets': {'smithi029.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHhvEccHXaj9KecRdhJ2R7k6deK/FnDjkxOTwaL94Ze0GkNEzZbozrPwNc2SMmJZq3gdDq6vTnghyy+IyRAxjQw=', 'smithi057.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBcE1vhLAy0ZftcWRW+Iu01zQaKILaNE+BFn/TIqb3zF05SMgiwusZ/3kJVVvXngHrx6SL2rEFn+kkOfgIxp3HY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'bed7681480ccb0192b91a35e691658867e1e96b7'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'bed7681480ccb0192b91a35e691658867e1e96b7'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'bed7681480ccb0192b91a35e691658867e1e96b7'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'f2de108beb0a6dde7f3e6c96039071828a1f6269', 'timestamp': '2023-12-01_18:01:25', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-02T02:47:46.180 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-12-02T02:47:46.187 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-12-02T02:47:46.206 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m0777 -- /var/run/ceph 2023-12-02T02:47:46.239 DEBUG:teuthology.orchestra.run.smithi057:> sudo install -d -m0777 -- /var/run/ceph 2023-12-02T02:47:46.283 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:47:46.283 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/scratch_devs of=/dev/stdout 2023-12-02T02:47:46.341 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-02T02:47:46.341 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_1 2023-12-02T02:47:46.399 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-02T02:47:46.399 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-02T02:47:46.399 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 665 Links: 1 2023-12-02T02:47:46.399 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-02T02:47:46.399 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2023-12-02T02:47:46.400 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2023-12-02 02:46:36.027881953 +0000 2023-12-02T02:47:46.400 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2023-12-02 02:42:15.567408028 +0000 2023-12-02T02:47:46.400 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2023-12-02 02:42:15.567408028 +0000 2023-12-02T02:47:46.400 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2023-12-02 02:42:15.567408028 +0000 2023-12-02T02:47:46.400 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-02T02:47:46.471 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2023-12-02T02:47:46.472 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2023-12-02T02:47:46.472 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000311892 s, 1.6 MB/s 2023-12-02T02:47:46.473 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-02T02:47:46.532 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_2 2023-12-02T02:47:46.589 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-02T02:47:46.589 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-02T02:47:46.589 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 683 Links: 1 2023-12-02T02:47:46.590 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-02T02:47:46.590 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2023-12-02T02:47:46.590 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2023-12-02 02:46:36.027881953 +0000 2023-12-02T02:47:46.590 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2023-12-02 02:42:16.096400391 +0000 2023-12-02T02:47:46.590 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2023-12-02 02:42:16.096400391 +0000 2023-12-02T02:47:46.590 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2023-12-02 02:42:16.096400391 +0000 2023-12-02T02:47:46.590 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-02T02:47:46.662 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2023-12-02T02:47:46.662 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2023-12-02T02:47:46.662 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000282159 s, 1.8 MB/s 2023-12-02T02:47:46.663 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-02T02:47:46.721 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_3 2023-12-02T02:47:46.778 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-02T02:47:46.778 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-02T02:47:46.778 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-12-02T02:47:46.778 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-02T02:47:46.779 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2023-12-02T02:47:46.779 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2023-12-02 02:46:36.027881953 +0000 2023-12-02T02:47:46.779 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2023-12-02 02:42:16.609392984 +0000 2023-12-02T02:47:46.779 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2023-12-02 02:42:16.609392984 +0000 2023-12-02T02:47:46.779 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2023-12-02 02:42:16.609392984 +0000 2023-12-02T02:47:46.779 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-02T02:47:46.851 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2023-12-02T02:47:46.851 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2023-12-02T02:47:46.851 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000324542 s, 1.6 MB/s 2023-12-02T02:47:46.852 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-02T02:47:46.910 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_4 2023-12-02T02:47:46.967 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-02T02:47:46.968 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-02T02:47:46.968 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 720 Links: 1 2023-12-02T02:47:46.968 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-02T02:47:46.968 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2023-12-02T02:47:46.968 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2023-12-02 02:46:36.028881938 +0000 2023-12-02T02:47:46.968 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2023-12-02 02:42:17.120385607 +0000 2023-12-02T02:47:46.968 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2023-12-02 02:42:17.120385607 +0000 2023-12-02T02:47:46.968 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2023-12-02 02:42:17.120385607 +0000 2023-12-02T02:47:46.968 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-02T02:47:47.040 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2023-12-02T02:47:47.040 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2023-12-02T02:47:47.040 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000317971 s, 1.6 MB/s 2023-12-02T02:47:47.042 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-02T02:47:47.100 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-02T02:47:47.100 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi057.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-02T02:47:47.100 INFO:tasks.ceph:Generating config... 2023-12-02T02:47:47.101 INFO:tasks.ceph:[client] debug rbd = 20 2023-12-02T02:47:47.101 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-12-02T02:47:47.101 INFO:tasks.ceph:[client] log to stderr = False 2023-12-02T02:47:47.101 INFO:tasks.ceph:[client] rbd default features = 125 2023-12-02T02:47:47.102 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2023-12-02T02:47:47.102 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T02:47:47.102 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T02:47:47.102 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T02:47:47.102 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T02:47:47.102 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T02:47:47.102 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T02:47:47.102 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T02:47:47.102 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T02:47:47.102 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-02T02:47:47.102 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-12-02T02:47:47.102 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-02T02:47:47.102 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-02T02:47:47.103 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-02T02:47:47.103 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-02T02:47:47.103 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-02T02:47:47.103 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-02T02:47:47.103 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2023-12-02T02:47:47.103 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2023-12-02T02:47:47.103 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-02T02:47:47.103 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-02T02:47:47.103 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-02T02:47:47.103 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-02T02:47:47.104 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-02T02:47:47.104 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-02T02:47:47.104 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-02T02:47:47.104 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-02T02:47:47.104 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-02T02:47:47.104 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-02T02:47:47.104 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-02T02:47:47.104 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-02T02:47:47.104 INFO:tasks.ceph:Setting up cluster2.mon.a... 2023-12-02T02:47:47.105 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2023-12-02T02:47:47.189 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/cluster2.keyring 2023-12-02T02:47:47.192 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2023-12-02T02:47:47.241 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-02T02:47:47.273 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.57')] 2023-12-02T02:47:47.273 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.57', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-12-02T02:47:47.274 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:47:47.274 DEBUG:teuthology.orchestra.run.smithi057:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-02T02:47:47.333 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.57 --print /home/ubuntu/cephtest/cluster2.monmap 2023-12-02T02:47:47.412 INFO:teuthology.orchestra.run.smithi057.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2023-12-02T02:47:47.412 INFO:teuthology.orchestra.run.smithi057.stdout:monmaptool: generated fsid 144563bd-d77d-446f-abf1-486ef977471e 2023-12-02T02:47:47.412 INFO:teuthology.orchestra.run.smithi057.stdout:setting min_mon_release = pacific 2023-12-02T02:47:47.412 INFO:teuthology.orchestra.run.smithi057.stdout:epoch 0 2023-12-02T02:47:47.412 INFO:teuthology.orchestra.run.smithi057.stdout:fsid 144563bd-d77d-446f-abf1-486ef977471e 2023-12-02T02:47:47.413 INFO:teuthology.orchestra.run.smithi057.stdout:last_changed 2023-12-02T02:47:47.410479+0000 2023-12-02T02:47:47.413 INFO:teuthology.orchestra.run.smithi057.stdout:created 2023-12-02T02:47:47.410479+0000 2023-12-02T02:47:47.413 INFO:teuthology.orchestra.run.smithi057.stdout:min_mon_release 16 (pacific) 2023-12-02T02:47:47.413 INFO:teuthology.orchestra.run.smithi057.stdout:election_strategy: 1 2023-12-02T02:47:47.413 INFO:teuthology.orchestra.run.smithi057.stdout:0: [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] mon.a 2023-12-02T02:47:47.413 INFO:teuthology.orchestra.run.smithi057.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2023-12-02T02:47:47.414 DEBUG:teuthology.orchestra.run.smithi057:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-02T02:47:47.469 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 144563bd-d77d-446f-abf1-486ef977471e... 2023-12-02T02:47:47.470 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-12-02T02:47:47.473 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-12-02T02:47:47.540 INFO:teuthology.orchestra.run.smithi029.stdout:[global] 2023-12-02T02:47:47.540 INFO:teuthology.orchestra.run.smithi029.stdout: chdir = "" 2023-12-02T02:47:47.540 INFO:teuthology.orchestra.run.smithi029.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-02T02:47:47.540 INFO:teuthology.orchestra.run.smithi029.stdout: auth supported = cephx 2023-12-02T02:47:47.540 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.540 INFO:teuthology.orchestra.run.smithi029.stdout: filestore xattr use omap = true 2023-12-02T02:47:47.540 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.541 INFO:teuthology.orchestra.run.smithi029.stdout: mon clock drift allowed = 1.000 2023-12-02T02:47:47.541 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.541 INFO:teuthology.orchestra.run.smithi029.stdout: osd crush chooseleaf type = 0 2023-12-02T02:47:47.541 INFO:teuthology.orchestra.run.smithi029.stdout: auth debug = true 2023-12-02T02:47:47.541 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.541 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on old message = true 2023-12-02T02:47:47.541 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on bug = true 2023-12-02T02:47:47.541 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.541 INFO:teuthology.orchestra.run.smithi029.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-02T02:47:47.541 INFO:teuthology.orchestra.run.smithi029.stdout: mon pg warn max object skew = 0 2023-12-02T02:47:47.541 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.541 INFO:teuthology.orchestra.run.smithi029.stdout: # disable pg_autoscaler by default for new pools 2023-12-02T02:47:47.541 INFO:teuthology.orchestra.run.smithi029.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-02T02:47:47.541 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.542 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default size = 2 2023-12-02T02:47:47.542 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.542 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd allow primary affinity = true 2023-12-02T02:47:47.542 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd allow pg remap = true 2023-12-02T02:47:47.542 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on legacy crush tunables = false 2023-12-02T02:47:47.542 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on crush straw calc version zero = false 2023-12-02T02:47:47.542 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on no sortbitwise = false 2023-12-02T02:47:47.542 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on osd down out interval zero = false 2023-12-02T02:47:47.542 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on too few osds = false 2023-12-02T02:47:47.542 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-02T02:47:47.542 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-02T02:47:47.542 INFO:teuthology.orchestra.run.smithi029.stdout: mon_allow_pool_size_one = true 2023-12-02T02:47:47.542 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.542 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-02T02:47:47.543 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.543 INFO:teuthology.orchestra.run.smithi029.stdout: osd default data pool replay window = 5 2023-12-02T02:47:47.543 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.543 INFO:teuthology.orchestra.run.smithi029.stdout: mon allow pool delete = true 2023-12-02T02:47:47.543 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.543 INFO:teuthology.orchestra.run.smithi029.stdout: mon cluster log file level = debug 2023-12-02T02:47:47.543 INFO:teuthology.orchestra.run.smithi029.stdout: debug asserts on shutdown = true 2023-12-02T02:47:47.543 INFO:teuthology.orchestra.run.smithi029.stdout: mon health detail to clog = false 2023-12-02T02:47:47.543 INFO:teuthology.orchestra.run.smithi029.stdout: mon host = 172.21.15.57 2023-12-02T02:47:47.543 INFO:teuthology.orchestra.run.smithi029.stdout: mon client directed command retry = 5 2023-12-02T02:47:47.543 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on pool no app = False 2023-12-02T02:47:47.543 INFO:teuthology.orchestra.run.smithi029.stdout: ms inject socket failures = 5000 2023-12-02T02:47:47.543 INFO:teuthology.orchestra.run.smithi029.stdout: fsid = 144563bd-d77d-446f-abf1-486ef977471e 2023-12-02T02:47:47.545 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.545 INFO:teuthology.orchestra.run.smithi029.stdout:[osd] 2023-12-02T02:47:47.545 INFO:teuthology.orchestra.run.smithi029.stdout: osd journal size = 100 2023-12-02T02:47:47.545 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.545 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub load threshold = 5.0 2023-12-02T02:47:47.545 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub max interval = 600 2023-12-02T02:47:47.545 INFO:teuthology.orchestra.run.smithi029.stdout: osd mclock profile = high_recovery_ops 2023-12-02T02:47:47.545 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.545 INFO:teuthology.orchestra.run.smithi029.stdout: osd recover clone overlap = true 2023-12-02T02:47:47.545 INFO:teuthology.orchestra.run.smithi029.stdout: osd recovery max chunk = 1048576 2023-12-02T02:47:47.545 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.545 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug shutdown = true 2023-12-02T02:47:47.546 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug op order = true 2023-12-02T02:47:47.546 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug verify stray on activate = true 2023-12-02T02:47:47.546 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug trim objects = true 2023-12-02T02:47:47.546 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.546 INFO:teuthology.orchestra.run.smithi029.stdout: osd open classes on start = true 2023-12-02T02:47:47.546 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug pg log writeout = true 2023-12-02T02:47:47.546 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.546 INFO:teuthology.orchestra.run.smithi029.stdout: osd deep scrub update digest min age = 30 2023-12-02T02:47:47.546 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.546 INFO:teuthology.orchestra.run.smithi029.stdout: osd map max advance = 10 2023-12-02T02:47:47.546 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.546 INFO:teuthology.orchestra.run.smithi029.stdout: journal zero on create = true 2023-12-02T02:47:47.546 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.546 INFO:teuthology.orchestra.run.smithi029.stdout: filestore ondisk finisher threads = 3 2023-12-02T02:47:47.547 INFO:teuthology.orchestra.run.smithi029.stdout: filestore apply finisher threads = 3 2023-12-02T02:47:47.547 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.547 INFO:teuthology.orchestra.run.smithi029.stdout: bdev debug aio = true 2023-12-02T02:47:47.547 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug misdirected ops = true 2023-12-02T02:47:47.547 INFO:teuthology.orchestra.run.smithi029.stdout: bluefs allocator = hybrid 2023-12-02T02:47:47.547 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore allocator = hybrid 2023-12-02T02:47:47.547 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore block size = 96636764160 2023-12-02T02:47:47.547 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore fsck on mount = True 2023-12-02T02:47:47.547 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluefs = 1/20 2023-12-02T02:47:47.547 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluestore = 1/20 2023-12-02T02:47:47.547 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-12-02T02:47:47.547 INFO:teuthology.orchestra.run.smithi029.stdout: debug osd = 20 2023-12-02T02:47:47.547 INFO:teuthology.orchestra.run.smithi029.stdout: debug rocksdb = 4/10 2023-12-02T02:47:47.547 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd backfillfull_ratio = 0.85 2023-12-02T02:47:47.548 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd full ratio = 0.9 2023-12-02T02:47:47.548 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd nearfull ratio = 0.8 2023-12-02T02:47:47.548 INFO:teuthology.orchestra.run.smithi029.stdout: osd failsafe full ratio = 0.95 2023-12-02T02:47:47.548 INFO:teuthology.orchestra.run.smithi029.stdout: osd objectstore = bluestore 2023-12-02T02:47:47.548 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.548 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr] 2023-12-02T02:47:47.548 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-12-02T02:47:47.548 INFO:teuthology.orchestra.run.smithi029.stdout: debug mgr = 20 2023-12-02T02:47:47.548 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 20 2023-12-02T02:47:47.548 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 20 2023-12-02T02:47:47.548 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min pgs per osd = 4 2023-12-02T02:47:47.548 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min bytes per osd = 10 2023-12-02T02:47:47.548 INFO:teuthology.orchestra.run.smithi029.stdout: mgr/telemetry/nag = false 2023-12-02T02:47:47.548 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.548 INFO:teuthology.orchestra.run.smithi029.stdout:[mon] 2023-12-02T02:47:47.549 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-12-02T02:47:47.549 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 20 2023-12-02T02:47:47.549 INFO:teuthology.orchestra.run.smithi029.stdout: debug paxos = 20 2023-12-02T02:47:47.549 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 20 2023-12-02T02:47:47.549 INFO:teuthology.orchestra.run.smithi029.stdout: mon data avail warn = 5 2023-12-02T02:47:47.549 INFO:teuthology.orchestra.run.smithi029.stdout: mon mgr mkfs grace = 240 2023-12-02T02:47:47.549 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min pgs per osd = 4 2023-12-02T02:47:47.549 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd reporter subtree level = osd 2023-12-02T02:47:47.549 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd prime pg temp = true 2023-12-02T02:47:47.549 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min bytes per osd = 10 2023-12-02T02:47:47.549 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.549 INFO:teuthology.orchestra.run.smithi029.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-02T02:47:47.549 INFO:teuthology.orchestra.run.smithi029.stdout: auth mon ticket ttl = 660 # 11m 2023-12-02T02:47:47.549 INFO:teuthology.orchestra.run.smithi029.stdout: auth service ticket ttl = 240 # 4m 2023-12-02T02:47:47.550 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.550 INFO:teuthology.orchestra.run.smithi029.stdout: # don't complain about insecure global_id in the test suite 2023-12-02T02:47:47.550 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-02T02:47:47.550 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-02T02:47:47.550 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.550 INFO:teuthology.orchestra.run.smithi029.stdout: # 1m isn't quite enough 2023-12-02T02:47:47.550 INFO:teuthology.orchestra.run.smithi029.stdout: mon_down_mkfs_grace = 2m 2023-12-02T02:47:47.550 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.550 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_filestore_osds = false 2023-12-02T02:47:47.550 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-12-02T02:47:47.550 INFO:teuthology.orchestra.run.smithi029.stdout:[client] 2023-12-02T02:47:47.550 INFO:teuthology.orchestra.run.smithi029.stdout: rgw cache enabled = true 2023-12-02T02:47:47.550 INFO:teuthology.orchestra.run.smithi029.stdout: rgw enable ops log = true 2023-12-02T02:47:47.551 INFO:teuthology.orchestra.run.smithi029.stdout: rgw enable usage log = true 2023-12-02T02:47:47.551 INFO:teuthology.orchestra.run.smithi029.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-02T02:47:47.551 INFO:teuthology.orchestra.run.smithi029.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-02T02:47:47.551 INFO:teuthology.orchestra.run.smithi029.stdout: debug rbd = 20 2023-12-02T02:47:47.551 INFO:teuthology.orchestra.run.smithi029.stdout: debug rbd_mirror = 15 2023-12-02T02:47:47.551 INFO:teuthology.orchestra.run.smithi029.stdout: log to stderr = False 2023-12-02T02:47:47.551 INFO:teuthology.orchestra.run.smithi029.stdout: rbd default features = 125 2023-12-02T02:47:47.551 INFO:teuthology.orchestra.run.smithi029.stdout: rbd mirror image policy type = simple 2023-12-02T02:47:47.551 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.a] 2023-12-02T02:47:47.551 INFO:teuthology.orchestra.run.smithi029.stdout:[client.mirror.0] 2023-12-02T02:47:47.551 INFO:teuthology.orchestra.run.smithi029.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T02:47:47.551 INFO:teuthology.orchestra.run.smithi029.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T02:47:47.551 INFO:teuthology.orchestra.run.smithi029.stdout:[client.mirror.1] 2023-12-02T02:47:47.551 INFO:teuthology.orchestra.run.smithi029.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T02:47:47.551 INFO:teuthology.orchestra.run.smithi029.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T02:47:47.552 INFO:teuthology.orchestra.run.smithi029.stdout:[client.mirror.2] 2023-12-02T02:47:47.552 INFO:teuthology.orchestra.run.smithi029.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T02:47:47.552 INFO:teuthology.orchestra.run.smithi029.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T02:47:47.552 INFO:teuthology.orchestra.run.smithi029.stdout:[client.mirror.3] 2023-12-02T02:47:47.552 INFO:teuthology.orchestra.run.smithi029.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T02:47:47.552 INFO:teuthology.orchestra.run.smithi029.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T02:47:47.570 INFO:teuthology.orchestra.run.smithi057.stdout:[global] 2023-12-02T02:47:47.570 INFO:teuthology.orchestra.run.smithi057.stdout: chdir = "" 2023-12-02T02:47:47.570 INFO:teuthology.orchestra.run.smithi057.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-02T02:47:47.570 INFO:teuthology.orchestra.run.smithi057.stdout: auth supported = cephx 2023-12-02T02:47:47.570 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.571 INFO:teuthology.orchestra.run.smithi057.stdout: filestore xattr use omap = true 2023-12-02T02:47:47.571 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.571 INFO:teuthology.orchestra.run.smithi057.stdout: mon clock drift allowed = 1.000 2023-12-02T02:47:47.571 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.571 INFO:teuthology.orchestra.run.smithi057.stdout: osd crush chooseleaf type = 0 2023-12-02T02:47:47.571 INFO:teuthology.orchestra.run.smithi057.stdout: auth debug = true 2023-12-02T02:47:47.571 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.571 INFO:teuthology.orchestra.run.smithi057.stdout: ms die on old message = true 2023-12-02T02:47:47.571 INFO:teuthology.orchestra.run.smithi057.stdout: ms die on bug = true 2023-12-02T02:47:47.571 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.571 INFO:teuthology.orchestra.run.smithi057.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-02T02:47:47.571 INFO:teuthology.orchestra.run.smithi057.stdout: mon pg warn max object skew = 0 2023-12-02T02:47:47.571 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.571 INFO:teuthology.orchestra.run.smithi057.stdout: # disable pg_autoscaler by default for new pools 2023-12-02T02:47:47.571 INFO:teuthology.orchestra.run.smithi057.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-02T02:47:47.572 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.572 INFO:teuthology.orchestra.run.smithi057.stdout: osd pool default size = 2 2023-12-02T02:47:47.572 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.572 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd allow primary affinity = true 2023-12-02T02:47:47.572 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd allow pg remap = true 2023-12-02T02:47:47.572 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on legacy crush tunables = false 2023-12-02T02:47:47.572 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on crush straw calc version zero = false 2023-12-02T02:47:47.572 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on no sortbitwise = false 2023-12-02T02:47:47.572 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on osd down out interval zero = false 2023-12-02T02:47:47.572 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on too few osds = false 2023-12-02T02:47:47.572 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-02T02:47:47.572 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-02T02:47:47.572 INFO:teuthology.orchestra.run.smithi057.stdout: mon_allow_pool_size_one = true 2023-12-02T02:47:47.572 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.573 INFO:teuthology.orchestra.run.smithi057.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-02T02:47:47.573 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.573 INFO:teuthology.orchestra.run.smithi057.stdout: osd default data pool replay window = 5 2023-12-02T02:47:47.573 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.573 INFO:teuthology.orchestra.run.smithi057.stdout: mon allow pool delete = true 2023-12-02T02:47:47.573 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.573 INFO:teuthology.orchestra.run.smithi057.stdout: mon cluster log file level = debug 2023-12-02T02:47:47.573 INFO:teuthology.orchestra.run.smithi057.stdout: debug asserts on shutdown = true 2023-12-02T02:47:47.573 INFO:teuthology.orchestra.run.smithi057.stdout: mon health detail to clog = false 2023-12-02T02:47:47.573 INFO:teuthology.orchestra.run.smithi057.stdout: mon host = 172.21.15.57 2023-12-02T02:47:47.573 INFO:teuthology.orchestra.run.smithi057.stdout: mon client directed command retry = 5 2023-12-02T02:47:47.574 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on pool no app = False 2023-12-02T02:47:47.574 INFO:teuthology.orchestra.run.smithi057.stdout: ms inject socket failures = 5000 2023-12-02T02:47:47.574 INFO:teuthology.orchestra.run.smithi057.stdout: fsid = 144563bd-d77d-446f-abf1-486ef977471e 2023-12-02T02:47:47.574 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.574 INFO:teuthology.orchestra.run.smithi057.stdout:[osd] 2023-12-02T02:47:47.574 INFO:teuthology.orchestra.run.smithi057.stdout: osd journal size = 100 2023-12-02T02:47:47.574 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.574 INFO:teuthology.orchestra.run.smithi057.stdout: osd scrub load threshold = 5.0 2023-12-02T02:47:47.574 INFO:teuthology.orchestra.run.smithi057.stdout: osd scrub max interval = 600 2023-12-02T02:47:47.574 INFO:teuthology.orchestra.run.smithi057.stdout: osd mclock profile = high_recovery_ops 2023-12-02T02:47:47.574 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.574 INFO:teuthology.orchestra.run.smithi057.stdout: osd recover clone overlap = true 2023-12-02T02:47:47.575 INFO:teuthology.orchestra.run.smithi057.stdout: osd recovery max chunk = 1048576 2023-12-02T02:47:47.575 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.575 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug shutdown = true 2023-12-02T02:47:47.575 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug op order = true 2023-12-02T02:47:47.575 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug verify stray on activate = true 2023-12-02T02:47:47.575 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug trim objects = true 2023-12-02T02:47:47.575 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.575 INFO:teuthology.orchestra.run.smithi057.stdout: osd open classes on start = true 2023-12-02T02:47:47.575 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug pg log writeout = true 2023-12-02T02:47:47.575 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.575 INFO:teuthology.orchestra.run.smithi057.stdout: osd deep scrub update digest min age = 30 2023-12-02T02:47:47.575 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.575 INFO:teuthology.orchestra.run.smithi057.stdout: osd map max advance = 10 2023-12-02T02:47:47.575 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.576 INFO:teuthology.orchestra.run.smithi057.stdout: journal zero on create = true 2023-12-02T02:47:47.576 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.576 INFO:teuthology.orchestra.run.smithi057.stdout: filestore ondisk finisher threads = 3 2023-12-02T02:47:47.576 INFO:teuthology.orchestra.run.smithi057.stdout: filestore apply finisher threads = 3 2023-12-02T02:47:47.576 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.576 INFO:teuthology.orchestra.run.smithi057.stdout: bdev debug aio = true 2023-12-02T02:47:47.576 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug misdirected ops = true 2023-12-02T02:47:47.576 INFO:teuthology.orchestra.run.smithi057.stdout: bluefs allocator = hybrid 2023-12-02T02:47:47.576 INFO:teuthology.orchestra.run.smithi057.stdout: bluestore allocator = hybrid 2023-12-02T02:47:47.576 INFO:teuthology.orchestra.run.smithi057.stdout: bluestore block size = 96636764160 2023-12-02T02:47:47.576 INFO:teuthology.orchestra.run.smithi057.stdout: bluestore fsck on mount = True 2023-12-02T02:47:47.577 INFO:teuthology.orchestra.run.smithi057.stdout: debug bluefs = 1/20 2023-12-02T02:47:47.577 INFO:teuthology.orchestra.run.smithi057.stdout: debug bluestore = 1/20 2023-12-02T02:47:47.577 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2023-12-02T02:47:47.577 INFO:teuthology.orchestra.run.smithi057.stdout: debug osd = 20 2023-12-02T02:47:47.577 INFO:teuthology.orchestra.run.smithi057.stdout: debug rocksdb = 4/10 2023-12-02T02:47:47.577 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd backfillfull_ratio = 0.85 2023-12-02T02:47:47.577 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd full ratio = 0.9 2023-12-02T02:47:47.577 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd nearfull ratio = 0.8 2023-12-02T02:47:47.577 INFO:teuthology.orchestra.run.smithi057.stdout: osd failsafe full ratio = 0.95 2023-12-02T02:47:47.577 INFO:teuthology.orchestra.run.smithi057.stdout: osd objectstore = bluestore 2023-12-02T02:47:47.577 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.577 INFO:teuthology.orchestra.run.smithi057.stdout:[mgr] 2023-12-02T02:47:47.577 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2023-12-02T02:47:47.577 INFO:teuthology.orchestra.run.smithi057.stdout: debug mgr = 20 2023-12-02T02:47:47.578 INFO:teuthology.orchestra.run.smithi057.stdout: debug mon = 20 2023-12-02T02:47:47.578 INFO:teuthology.orchestra.run.smithi057.stdout: debug auth = 20 2023-12-02T02:47:47.578 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min pgs per osd = 4 2023-12-02T02:47:47.578 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min bytes per osd = 10 2023-12-02T02:47:47.578 INFO:teuthology.orchestra.run.smithi057.stdout: mgr/telemetry/nag = false 2023-12-02T02:47:47.578 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.578 INFO:teuthology.orchestra.run.smithi057.stdout:[mon] 2023-12-02T02:47:47.578 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2023-12-02T02:47:47.578 INFO:teuthology.orchestra.run.smithi057.stdout: debug mon = 20 2023-12-02T02:47:47.578 INFO:teuthology.orchestra.run.smithi057.stdout: debug paxos = 20 2023-12-02T02:47:47.578 INFO:teuthology.orchestra.run.smithi057.stdout: debug auth = 20 2023-12-02T02:47:47.578 INFO:teuthology.orchestra.run.smithi057.stdout: mon data avail warn = 5 2023-12-02T02:47:47.579 INFO:teuthology.orchestra.run.smithi057.stdout: mon mgr mkfs grace = 240 2023-12-02T02:47:47.579 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min pgs per osd = 4 2023-12-02T02:47:47.579 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd reporter subtree level = osd 2023-12-02T02:47:47.579 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd prime pg temp = true 2023-12-02T02:47:47.579 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min bytes per osd = 10 2023-12-02T02:47:47.579 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.579 INFO:teuthology.orchestra.run.smithi057.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-02T02:47:47.579 INFO:teuthology.orchestra.run.smithi057.stdout: auth mon ticket ttl = 660 # 11m 2023-12-02T02:47:47.579 INFO:teuthology.orchestra.run.smithi057.stdout: auth service ticket ttl = 240 # 4m 2023-12-02T02:47:47.579 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.579 INFO:teuthology.orchestra.run.smithi057.stdout: # don't complain about insecure global_id in the test suite 2023-12-02T02:47:47.579 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-02T02:47:47.579 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-02T02:47:47.579 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.579 INFO:teuthology.orchestra.run.smithi057.stdout: # 1m isn't quite enough 2023-12-02T02:47:47.580 INFO:teuthology.orchestra.run.smithi057.stdout: mon_down_mkfs_grace = 2m 2023-12-02T02:47:47.580 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.580 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_filestore_osds = false 2023-12-02T02:47:47.580 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:47:47.580 INFO:teuthology.orchestra.run.smithi057.stdout:[client] 2023-12-02T02:47:47.580 INFO:teuthology.orchestra.run.smithi057.stdout: rgw cache enabled = true 2023-12-02T02:47:47.580 INFO:teuthology.orchestra.run.smithi057.stdout: rgw enable ops log = true 2023-12-02T02:47:47.580 INFO:teuthology.orchestra.run.smithi057.stdout: rgw enable usage log = true 2023-12-02T02:47:47.580 INFO:teuthology.orchestra.run.smithi057.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-02T02:47:47.580 INFO:teuthology.orchestra.run.smithi057.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-02T02:47:47.580 INFO:teuthology.orchestra.run.smithi057.stdout: debug rbd = 20 2023-12-02T02:47:47.580 INFO:teuthology.orchestra.run.smithi057.stdout: debug rbd_mirror = 15 2023-12-02T02:47:47.580 INFO:teuthology.orchestra.run.smithi057.stdout: log to stderr = False 2023-12-02T02:47:47.580 INFO:teuthology.orchestra.run.smithi057.stdout: rbd default features = 125 2023-12-02T02:47:47.581 INFO:teuthology.orchestra.run.smithi057.stdout: rbd mirror image policy type = simple 2023-12-02T02:47:47.581 INFO:teuthology.orchestra.run.smithi057.stdout:[mon.a] 2023-12-02T02:47:47.581 INFO:teuthology.orchestra.run.smithi057.stdout:[client.mirror.0] 2023-12-02T02:47:47.581 INFO:teuthology.orchestra.run.smithi057.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T02:47:47.581 INFO:teuthology.orchestra.run.smithi057.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T02:47:47.581 INFO:teuthology.orchestra.run.smithi057.stdout:[client.mirror.1] 2023-12-02T02:47:47.581 INFO:teuthology.orchestra.run.smithi057.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T02:47:47.581 INFO:teuthology.orchestra.run.smithi057.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T02:47:47.581 INFO:teuthology.orchestra.run.smithi057.stdout:[client.mirror.2] 2023-12-02T02:47:47.581 INFO:teuthology.orchestra.run.smithi057.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T02:47:47.581 INFO:teuthology.orchestra.run.smithi057.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T02:47:47.581 INFO:teuthology.orchestra.run.smithi057.stdout:[client.mirror.3] 2023-12-02T02:47:47.581 INFO:teuthology.orchestra.run.smithi057.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-02T02:47:47.581 INFO:teuthology.orchestra.run.smithi057.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-02T02:47:47.587 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2023-12-02T02:47:47.588 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2023-12-02T02:47:47.687 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-02T02:47:47.688 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:47:47.688 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2023-12-02T02:47:47.702 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:47:47.702 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2023-12-02T02:47:47.757 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi029.front.sepia.ceph.com 2023-12-02T02:47:47.757 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-02T02:47:47.757 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/cluster2.keyring 2023-12-02T02:47:47.757 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-02T02:47:47.811 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-02T02:47:47.811 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-12-02T02:47:47.867 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi057.front.sepia.ceph.com 2023-12-02T02:47:47.867 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:47:47.867 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/ceph/cluster2.keyring 2023-12-02T02:47:47.868 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-02T02:47:47.914 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:47:47.914 DEBUG:teuthology.orchestra.run.smithi057:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-12-02T02:47:47.969 INFO:tasks.ceph:Setting up mon nodes... 2023-12-02T02:47:47.970 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-02T02:47:47.970 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2023-12-02T02:47:48.036 INFO:teuthology.orchestra.run.smithi029.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2023-12-02T02:47:48.038 INFO:tasks.ceph:Setting up mds nodes... 2023-12-02T02:47:48.042 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-02T02:47:48.042 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2023-12-02T02:47:48.090 INFO:teuthology.orchestra.run.smithi029.stdout:creating /etc/ceph/cluster2.client.0.keyring 2023-12-02T02:47:48.110 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2023-12-02T02:47:48.156 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2023-12-02T02:47:48.175 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2023-12-02T02:47:48.220 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2023-12-02T02:47:48.239 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2023-12-02T02:47:48.285 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2023-12-02T02:47:48.303 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2023-12-02T02:47:48.349 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2023-12-02T02:47:48.367 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2023-12-02T02:47:48.413 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2023-12-02T02:47:48.431 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2023-12-02T02:47:48.477 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2023-12-02T02:47:48.496 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2023-12-02T02:47:48.541 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2023-12-02T02:47:48.560 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2023-12-02T02:47:48.606 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2023-12-02T02:47:48.624 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-02T02:47:48.624 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi029.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi057.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-02T02:47:48.624 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2023-12-02T02:47:48.654 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-02T02:47:48.654 INFO:tasks.ceph:role: cluster2.osd.0 2023-12-02T02:47:48.655 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi057.front.sepia.ceph.com 2023-12-02T02:47:48.655 DEBUG:teuthology.orchestra.run.smithi057:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-02T02:47:48.753 INFO:teuthology.orchestra.run.smithi057.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-02T02:47:48.753 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-02T02:47:48.753 INFO:teuthology.orchestra.run.smithi057.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-02T02:47:48.753 INFO:teuthology.orchestra.run.smithi057.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-02T02:47:48.753 INFO:teuthology.orchestra.run.smithi057.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-02T02:47:48.753 INFO:teuthology.orchestra.run.smithi057.stdout: = sunit=0 swidth=0 blks 2023-12-02T02:47:48.753 INFO:teuthology.orchestra.run.smithi057.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-02T02:47:48.753 INFO:teuthology.orchestra.run.smithi057.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-02T02:47:48.753 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-02T02:47:48.753 INFO:teuthology.orchestra.run.smithi057.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-02T02:47:50.097 INFO:teuthology.orchestra.run.smithi057.stdout:Discarding blocks...Done. 2023-12-02T02:47:50.100 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi057.front.sepia.ceph.com -o noatime 2023-12-02T02:47:50.100 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2023-12-02T02:47:50.143 DEBUG:teuthology.orchestra.run.smithi057:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2023-12-02T02:47:50.223 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2023-12-02T02:47:50.297 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-02T02:47:50.297 INFO:tasks.ceph:role: cluster2.osd.1 2023-12-02T02:47:50.297 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi057.front.sepia.ceph.com 2023-12-02T02:47:50.297 DEBUG:teuthology.orchestra.run.smithi057:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-02T02:47:50.398 INFO:teuthology.orchestra.run.smithi057.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-02T02:47:50.398 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-02T02:47:50.398 INFO:teuthology.orchestra.run.smithi057.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-02T02:47:50.398 INFO:teuthology.orchestra.run.smithi057.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-02T02:47:50.398 INFO:teuthology.orchestra.run.smithi057.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-02T02:47:50.398 INFO:teuthology.orchestra.run.smithi057.stdout: = sunit=0 swidth=0 blks 2023-12-02T02:47:50.398 INFO:teuthology.orchestra.run.smithi057.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-02T02:47:50.399 INFO:teuthology.orchestra.run.smithi057.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-02T02:47:50.399 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-02T02:47:50.399 INFO:teuthology.orchestra.run.smithi057.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-02T02:47:51.710 INFO:teuthology.orchestra.run.smithi057.stdout:Discarding blocks...Done. 2023-12-02T02:47:51.713 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi057.front.sepia.ceph.com -o noatime 2023-12-02T02:47:51.713 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2023-12-02T02:47:51.759 DEBUG:teuthology.orchestra.run.smithi057:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2023-12-02T02:47:51.796 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2023-12-02T02:47:51.868 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-02T02:47:51.868 INFO:tasks.ceph:role: cluster2.osd.2 2023-12-02T02:47:51.869 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi057.front.sepia.ceph.com 2023-12-02T02:47:51.869 DEBUG:teuthology.orchestra.run.smithi057:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-02T02:47:51.967 INFO:teuthology.orchestra.run.smithi057.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-02T02:47:51.967 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-02T02:47:51.968 INFO:teuthology.orchestra.run.smithi057.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-02T02:47:51.968 INFO:teuthology.orchestra.run.smithi057.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-02T02:47:51.968 INFO:teuthology.orchestra.run.smithi057.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-02T02:47:51.968 INFO:teuthology.orchestra.run.smithi057.stdout: = sunit=0 swidth=0 blks 2023-12-02T02:47:51.968 INFO:teuthology.orchestra.run.smithi057.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-02T02:47:51.968 INFO:teuthology.orchestra.run.smithi057.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-02T02:47:51.968 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-02T02:47:51.968 INFO:teuthology.orchestra.run.smithi057.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-02T02:47:53.280 INFO:teuthology.orchestra.run.smithi057.stdout:Discarding blocks...Done. 2023-12-02T02:47:53.283 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi057.front.sepia.ceph.com -o noatime 2023-12-02T02:47:53.283 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2023-12-02T02:47:53.328 DEBUG:teuthology.orchestra.run.smithi057:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2023-12-02T02:47:53.366 DEBUG:teuthology.orchestra.run.smithi057:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-02T02:47:53.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-12-02T02:47:53.467+0000 7feab3418500 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2023-12-02T02:47:53.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-12-02T02:47:53.468+0000 7feab3418500 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2023-12-02T02:47:53.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-12-02T02:47:53.469+0000 7feab3418500 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-02T02:47:53.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-12-02T02:47:53.469+0000 7feab3418500 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-02T02:47:53.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-12-02T02:47:53.469+0000 7feab3418500 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-02T02:47:53.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-12-02T02:47:53.470+0000 7feab3418500 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2023-12-02T02:47:56.346 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2023-12-02T02:47:56.382 DEBUG:teuthology.orchestra.run.smithi057:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-02T02:47:56.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-12-02T02:47:56.473+0000 7f327aa15500 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2023-12-02T02:47:56.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-12-02T02:47:56.473+0000 7f327aa15500 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2023-12-02T02:47:56.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-12-02T02:47:56.473+0000 7f327aa15500 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-02T02:47:56.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-12-02T02:47:56.473+0000 7f327aa15500 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-02T02:47:56.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-12-02T02:47:56.473+0000 7f327aa15500 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-02T02:47:56.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-12-02T02:47:56.474+0000 7f327aa15500 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2023-12-02T02:47:59.342 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2023-12-02T02:47:59.377 DEBUG:teuthology.orchestra.run.smithi057:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-02T02:47:59.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-12-02T02:47:59.470+0000 7f1874594500 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2023-12-02T02:47:59.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-12-02T02:47:59.470+0000 7f1874594500 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2023-12-02T02:47:59.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-12-02T02:47:59.470+0000 7f1874594500 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-02T02:47:59.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-12-02T02:47:59.470+0000 7f1874594500 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-02T02:47:59.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-12-02T02:47:59.470+0000 7f1874594500 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-02T02:47:59.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-12-02T02:47:59.471+0000 7f1874594500 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2023-12-02T02:48:02.352 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2023-12-02T02:48:02.387 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-02T02:48:02.387 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-02T02:48:02.387 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2023-12-02T02:48:02.425 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:48:02.425 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2023-12-02T02:48:02.460 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:48:02.461 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2023-12-02T02:48:02.533 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:48:02.533 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2023-12-02T02:48:02.606 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-12-02T02:48:02.606 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2023-12-02T02:48:02.625 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:48:02.625 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2023-12-02T02:48:02.662 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:48:02.663 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2023-12-02T02:48:02.718 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:48:02.718 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2023-12-02T02:48:02.775 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:48:02.775 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2023-12-02T02:48:02.831 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:48:02.831 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2023-12-02T02:48:02.887 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:48:02.887 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2023-12-02T02:48:02.942 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:48:02.943 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2023-12-02T02:48:02.999 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:48:02.999 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2023-12-02T02:48:03.055 INFO:tasks.ceph:Adding keys to all mons... 2023-12-02T02:48:03.055 DEBUG:teuthology.orchestra.run.smithi057:> sudo tee -a /etc/ceph/cluster2.keyring 2023-12-02T02:48:03.125 INFO:teuthology.orchestra.run.smithi057.stdout:[mgr.x] 2023-12-02T02:48:03.125 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDUmmplP9gHAhAA3X0xNVgNnAdsAn6F6oGwjw== 2023-12-02T02:48:03.125 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.0] 2023-12-02T02:48:03.125 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDZmmplpDTtGxAAtMXxzHO7HU3L88Qiije2nw== 2023-12-02T02:48:03.125 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.1] 2023-12-02T02:48:03.125 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDcmmpl9BZQHBAAgLayMJOHfnQU4S4yVnzc0Q== 2023-12-02T02:48:03.125 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.2] 2023-12-02T02:48:03.125 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDfmmpld4YaHBAAsGPEBLM0LUJc0VxwK3DLSw== 2023-12-02T02:48:03.125 INFO:teuthology.orchestra.run.smithi057.stdout:[client.0] 2023-12-02T02:48:03.125 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDUmmplDcpHBRAAy2OtXkjnCuoPsiMPueuNKw== 2023-12-02T02:48:03.125 INFO:teuthology.orchestra.run.smithi057.stdout:[client.mirror] 2023-12-02T02:48:03.126 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDUmmplOzQ6CRAAUIjVJQJPSGS1GzrsKTYsbw== 2023-12-02T02:48:03.126 INFO:teuthology.orchestra.run.smithi057.stdout:[client.mirror.0] 2023-12-02T02:48:03.126 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDUmmplt/0JDRAAk9zJSwtvwxUyREdQ2ykhDA== 2023-12-02T02:48:03.126 INFO:teuthology.orchestra.run.smithi057.stdout:[client.mirror.1] 2023-12-02T02:48:03.126 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDUmmpl6uDiEBAAJraUZqpIoEk51eDpAxreUA== 2023-12-02T02:48:03.126 INFO:teuthology.orchestra.run.smithi057.stdout:[client.mirror.2] 2023-12-02T02:48:03.126 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDUmmplpo2xFBAAdVgxqWwmj4YzVeeCsJheGA== 2023-12-02T02:48:03.126 INFO:teuthology.orchestra.run.smithi057.stdout:[client.mirror.3] 2023-12-02T02:48:03.126 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDUmmplFwGHGBAAVCmykGSiXSCrVxCIpPAzjQ== 2023-12-02T02:48:03.126 INFO:teuthology.orchestra.run.smithi057.stdout:[client.mirror.4] 2023-12-02T02:48:03.126 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDUmmplvadbHBAALLsiYXgByE2AsWL++M8Q6w== 2023-12-02T02:48:03.126 INFO:teuthology.orchestra.run.smithi057.stdout:[client.mirror.5] 2023-12-02T02:48:03.126 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDUmmplr24qIBAAdmxh8wZOXWQitYhAJAm8Mw== 2023-12-02T02:48:03.126 INFO:teuthology.orchestra.run.smithi057.stdout:[client.mirror.6] 2023-12-02T02:48:03.127 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDUmmplUcIDJBAASv+SAViXKOWEKJyshODmyw== 2023-12-02T02:48:03.127 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-02T02:48:03.225 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-02T02:48:03.281 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-02T02:48:03.337 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-02T02:48:03.393 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T02:48:03.452 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T02:48:03.511 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T02:48:03.570 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T02:48:03.629 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T02:48:03.688 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T02:48:03.748 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T02:48:03.825 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T02:48:03.884 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-02T02:48:03.943 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-02T02:48:03.944 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2023-12-02T02:48:03.974 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2023-12-02T02:48:04.094 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2023-12-02T02:48:04.125 DEBUG:teuthology.orchestra.run.smithi057:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2023-12-02T02:48:04.180 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2023-12-02T02:48:04.181 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2023-12-02T02:48:04.181 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2023-12-02T02:48:04.222 INFO:tasks.ceph.cluster2.mon.a:Started 2023-12-02T02:48:04.222 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2023-12-02T02:48:04.223 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2023-12-02T02:48:04.223 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2023-12-02T02:48:04.225 INFO:tasks.ceph.cluster2.mgr.x:Started 2023-12-02T02:48:04.225 INFO:tasks.ceph:Setting crush tunables to default 2023-12-02T02:48:04.226 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster cluster2 osd crush tunables default 2023-12-02T02:48:04.449 INFO:teuthology.orchestra.run.smithi057.stderr:adjusted tunables profile to default 2023-12-02T02:48:04.460 INFO:tasks.ceph:check_enable_crimson: False 2023-12-02T02:48:04.460 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2023-12-02T02:48:04.461 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:48:04.461 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2023-12-02T02:48:04.491 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:48:04.492 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2023-12-02T02:48:04.562 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:48:04.562 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2023-12-02T02:48:04.632 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster cluster2 osd new d2adfb2e-8c08-4578-87f3-e8237f28394b 0 2023-12-02T02:48:04.720 INFO:tasks.ceph.cluster2.mgr.x.smithi029.stderr:2023-12-02T02:48:04.718+0000 7f1ab441b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-02T02:48:04.767 INFO:tasks.ceph.cluster2.mgr.x.smithi029.stderr:2023-12-02T02:48:04.765+0000 7f1ab441b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-02T02:48:04.862 INFO:teuthology.orchestra.run.smithi057.stdout:0 2023-12-02T02:48:04.873 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster cluster2 osd new 214ac73d-ce6a-458f-a609-23472e3dbc90 1 2023-12-02T02:48:05.059 INFO:teuthology.orchestra.run.smithi057.stdout:1 2023-12-02T02:48:05.070 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster cluster2 osd new 3bd602b2-3d2b-4b4a-9b27-598bf9731a17 2 2023-12-02T02:48:05.173 INFO:tasks.ceph.cluster2.mgr.x.smithi029.stderr:2023-12-02T02:48:05.170+0000 7f1ab441b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-02T02:48:05.218 INFO:tasks.ceph.cluster2.mgr.x.smithi029.stderr:2023-12-02T02:48:05.216+0000 7f1ab441b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-02T02:48:05.261 INFO:teuthology.orchestra.run.smithi057.stdout:2 2023-12-02T02:48:05.266 INFO:tasks.ceph.cluster2.mgr.x.smithi029.stderr:2023-12-02T02:48:05.260+0000 7f1ab441b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-02T02:48:05.271 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2023-12-02T02:48:05.272 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2023-12-02T02:48:05.274 INFO:tasks.ceph.cluster2.osd.0:Started 2023-12-02T02:48:05.274 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2023-12-02T02:48:05.274 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2023-12-02T02:48:05.276 INFO:tasks.ceph.cluster2.osd.1:Started 2023-12-02T02:48:05.276 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2023-12-02T02:48:05.277 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2023-12-02T02:48:05.278 INFO:tasks.ceph.cluster2.osd.2:Started 2023-12-02T02:48:05.279 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-02T02:48:05.316 INFO:tasks.ceph.cluster2.mgr.x.smithi029.stderr:2023-12-02T02:48:05.314+0000 7f1ab441b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-02T02:48:05.359 INFO:tasks.ceph.cluster2.mgr.x.smithi029.stderr:2023-12-02T02:48:05.357+0000 7f1ab441b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-02T02:48:05.460 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:48:05.460 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":5,"fsid":"144563bd-d77d-446f-abf1-486ef977471e","created":"2023-12-02T02:48:04.334855+0000","modified":"2023-12-02T02:48:05.253733+0000","last_up_change":"0.000000","last_in_change":"2023-12-02T02:48:05.253733+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d2adfb2e-8c08-4578-87f3-e8237f28394b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"214ac73d-ce6a-458f-a609-23472e3dbc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3bd602b2-3d2b-4b4a-9b27-598bf9731a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-02T02:48:05.462 INFO:tasks.ceph.cluster2.mgr.x.smithi029.stderr:2023-12-02T02:48:05.460+0000 7f1ab441b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-02T02:48:05.471 INFO:tasks.ceph.ceph_manager.cluster2:[] 2023-12-02T02:48:05.471 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-02T02:48:05.517 INFO:tasks.ceph.cluster2.mgr.x.smithi029.stderr:2023-12-02T02:48:05.515+0000 7f1ab441b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-02T02:48:05.561 INFO:tasks.ceph.cluster2.mgr.x.smithi029.stderr:2023-12-02T02:48:05.558+0000 7f1ab441b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-02T02:48:05.610 INFO:tasks.ceph.cluster2.mgr.x.smithi029.stderr:2023-12-02T02:48:05.608+0000 7f1ab441b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-02T02:48:05.655 INFO:tasks.ceph.cluster2.mgr.x.smithi029.stderr:2023-12-02T02:48:05.652+0000 7f1ab441b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-02T02:48:05.774 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-12-02T02:48:05.874 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr:2023-12-02T02:48:05.870+0000 7f60a9c18500 -1 Falling back to public interface 2023-12-02T02:48:05.878 INFO:tasks.ceph.cluster2.osd.1.smithi057.stderr:2023-12-02T02:48:05.874+0000 7fbdae010500 -1 Falling back to public interface 2023-12-02T02:48:05.878 INFO:tasks.ceph.cluster2.osd.2.smithi057.stderr:2023-12-02T02:48:05.874+0000 7f3487412500 -1 Falling back to public interface 2023-12-02T02:48:05.878 INFO:tasks.ceph.cluster2.osd.2.smithi057.stderr:2023-12-02T02:48:05.875+0000 7f3485d90640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6808/0: (98) Address already in use 2023-12-02T02:48:05.942 INFO:teuthology.misc.health.smithi057.stdout: 2023-12-02T02:48:05.942 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":5,"fsid":"144563bd-d77d-446f-abf1-486ef977471e","created":"2023-12-02T02:48:04.334855+0000","modified":"2023-12-02T02:48:05.253733+0000","last_up_change":"0.000000","last_in_change":"2023-12-02T02:48:05.253733+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d2adfb2e-8c08-4578-87f3-e8237f28394b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"214ac73d-ce6a-458f-a609-23472e3dbc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3bd602b2-3d2b-4b4a-9b27-598bf9731a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-02T02:48:05.951 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-02T02:48:05.983 INFO:tasks.ceph.cluster2.mgr.x.smithi029.stderr:2023-12-02T02:48:05.981+0000 7f1ab441b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-02T02:48:06.121 INFO:tasks.ceph.cluster2.mgr.x.smithi029.stderr:2023-12-02T02:48:06.118+0000 7f1ab441b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-02T02:48:06.216 INFO:tasks.ceph.cluster2.mgr.x.smithi029.stderr:2023-12-02T02:48:06.213+0000 7f1ab441b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-02T02:48:06.260 INFO:tasks.ceph.cluster2.mgr.x.smithi029.stderr:2023-12-02T02:48:06.257+0000 7f1ab441b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-02T02:48:06.360 INFO:tasks.ceph.cluster2.mgr.x.smithi029.stderr:2023-12-02T02:48:06.357+0000 7f1ab441b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-02T02:48:06.455 INFO:tasks.ceph.cluster2.mgr.x.smithi029.stderr:2023-12-02T02:48:06.452+0000 7f1ab441b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-02T02:48:06.597 INFO:tasks.ceph.cluster2.mgr.x.smithi029.stderr:2023-12-02T02:48:06.594+0000 7f1ab441b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-02T02:48:06.641 INFO:tasks.ceph.cluster2.mgr.x.smithi029.stderr:2023-12-02T02:48:06.638+0000 7f1ab441b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-02T02:48:06.750 INFO:tasks.ceph.cluster2.mgr.x.smithi029.stderr:2023-12-02T02:48:06.747+0000 7f1ab441b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-02T02:48:07.374 INFO:tasks.ceph.cluster2.mgr.x.smithi029.stderr:2023-12-02T02:48:07.371+0000 7f1ab441b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-02T02:48:07.648 INFO:tasks.ceph.cluster2.mgr.x.smithi029.stderr:2023-12-02T02:48:07.644+0000 7f1ab441b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-02T02:48:07.698 INFO:tasks.ceph.cluster2.mgr.x.smithi029.stderr:2023-12-02T02:48:07.695+0000 7f1ab441b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-02T02:48:07.803 INFO:tasks.ceph.cluster2.mgr.x.smithi029.stderr:2023-12-02T02:48:07.800+0000 7f1ab441b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-02T02:48:08.827 INFO:tasks.ceph.cluster2.mgr.x.smithi029.stderr:2023-12-02T02:48:08.824+0000 7f1a553cc640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-02T02:48:09.344 INFO:tasks.ceph.cluster2.osd.1.smithi057.stderr:2023-12-02T02:48:09.340+0000 7fbdae010500 -1 osd.1 0 log_to_monitors true 2023-12-02T02:48:09.385 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr:2023-12-02T02:48:09.382+0000 7f60a9c18500 -1 osd.0 0 log_to_monitors true 2023-12-02T02:48:09.387 INFO:tasks.ceph.cluster2.osd.2.smithi057.stderr:2023-12-02T02:48:09.384+0000 7f3487412500 -1 osd.2 0 log_to_monitors true 2023-12-02T02:48:12.254 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-12-02T02:48:12.684 INFO:teuthology.misc.health.smithi057.stdout: 2023-12-02T02:48:12.684 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":7,"fsid":"144563bd-d77d-446f-abf1-486ef977471e","created":"2023-12-02T02:48:04.334855+0000","modified":"2023-12-02T02:48:11.310012+0000","last_up_change":"0.000000","last_in_change":"2023-12-02T02:48:05.253733+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d2adfb2e-8c08-4578-87f3-e8237f28394b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"214ac73d-ce6a-458f-a609-23472e3dbc90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3bd602b2-3d2b-4b4a-9b27-598bf9731a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-02T02:48:12.702 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-02T02:48:13.162 INFO:tasks.ceph.cluster2.osd.1.smithi057.stderr:2023-12-02T02:48:13.159+0000 7fbdaa156640 -1 osd.1 0 waiting for initial osdmap 2023-12-02T02:48:13.179 INFO:tasks.ceph.cluster2.osd.1.smithi057.stderr:2023-12-02T02:48:13.176+0000 7fbda1768640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-02T02:48:13.211 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr:2023-12-02T02:48:13.208+0000 7f60a5676640 -1 osd.0 0 waiting for initial osdmap 2023-12-02T02:48:13.226 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr:2023-12-02T02:48:13.223+0000 7f609d49c640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-02T02:48:13.234 INFO:tasks.ceph.cluster2.osd.2.smithi057.stderr:2023-12-02T02:48:13.231+0000 7f3482d77640 -1 osd.2 0 waiting for initial osdmap 2023-12-02T02:48:13.247 INFO:tasks.ceph.cluster2.osd.2.smithi057.stderr:2023-12-02T02:48:13.245+0000 7f347ab9d640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-02T02:48:19.009 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-12-02T02:48:19.300 INFO:teuthology.misc.health.smithi057.stdout: 2023-12-02T02:48:19.300 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":12,"fsid":"144563bd-d77d-446f-abf1-486ef977471e","created":"2023-12-02T02:48:04.334855+0000","modified":"2023-12-02T02:48:18.346568+0000","last_up_change":"2023-12-02T02:48:14.161811+0000","last_in_change":"2023-12-02T02:48:05.253733+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-02T02:48:15.878427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d2adfb2e-8c08-4578-87f3-e8237f28394b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":3474823585},{"type":"v1","addr":"172.21.15.57:6801","nonce":3474823585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3474823585},{"type":"v1","addr":"172.21.15.57:6803","nonce":3474823585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3474823585},{"type":"v1","addr":"172.21.15.57:6807","nonce":3474823585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3474823585},{"type":"v1","addr":"172.21.15.57:6805","nonce":3474823585}]},"public_addr":"172.21.15.57:6801/3474823585","cluster_addr":"172.21.15.57:6803/3474823585","heartbeat_back_addr":"172.21.15.57:6807/3474823585","heartbeat_front_addr":"172.21.15.57:6805/3474823585","state":["exists","up"]},{"osd":1,"uuid":"214ac73d-ce6a-458f-a609-23472e3dbc90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3625761898},{"type":"v1","addr":"172.21.15.57:6810","nonce":3625761898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3625761898},{"type":"v1","addr":"172.21.15.57:6814","nonce":3625761898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6819","nonce":3625761898},{"type":"v1","addr":"172.21.15.57:6821","nonce":3625761898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3625761898},{"type":"v1","addr":"172.21.15.57:6818","nonce":3625761898}]},"public_addr":"172.21.15.57:6810/3625761898","cluster_addr":"172.21.15.57:6814/3625761898","heartbeat_back_addr":"172.21.15.57:6821/3625761898","heartbeat_front_addr":"172.21.15.57:6818/3625761898","state":["exists","up"]},{"osd":2,"uuid":"3bd602b2-3d2b-4b4a-9b27-598bf9731a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6809","nonce":467796665},{"type":"v1","addr":"172.21.15.57:6811","nonce":467796665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6813","nonce":467796665},{"type":"v1","addr":"172.21.15.57:6815","nonce":467796665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":467796665},{"type":"v1","addr":"172.21.15.57:6823","nonce":467796665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6817","nonce":467796665},{"type":"v1","addr":"172.21.15.57:6820","nonce":467796665}]},"public_addr":"172.21.15.57:6811/467796665","cluster_addr":"172.21.15.57:6815/467796665","heartbeat_back_addr":"172.21.15.57:6823/467796665","heartbeat_front_addr":"172.21.15.57:6820/467796665","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T02:48:10.375797+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T02:48:10.298688+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T02:48:10.355416+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-02T02:48:19.311 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-12-02T02:48:19.311 INFO:tasks.ceph:Creating RBD pool 2023-12-02T02:48:19.312 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster cluster2 osd pool create rbd 8 2023-12-02T02:48:20.356 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'rbd' created 2023-12-02T02:48:20.374 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-02T02:48:21.357 INFO:teuthology.orchestra.run.smithi057.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-02T02:48:21.377 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2023-12-02T02:48:21.378 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2023-12-02T02:48:21.378 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2023-12-02T02:48:21.378 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2023-12-02T02:48:21.378 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-02T02:48:21.726 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:48:21.738 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":5,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1900155724},{"type":"v1","addr":"172.21.15.29:6827","nonce":1900155724}]},"active_addr":"172.21.15.29:6827/1900155724","active_change":"2023-12-02T02:48:07.814165+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":1166603667}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":3834944185}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":1042398954}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":1241531809}]}]} 2023-12-02T02:48:21.740 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2023-12-02T02:48:21.740 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2023-12-02T02:48:21.740 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-02T02:48:22.036 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:48:22.036 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":14,"fsid":"144563bd-d77d-446f-abf1-486ef977471e","created":"2023-12-02T02:48:04.334855+0000","modified":"2023-12-02T02:48:21.354734+0000","last_up_change":"2023-12-02T02:48:14.161811+0000","last_in_change":"2023-12-02T02:48:05.253733+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-02T02:48:15.878427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-02T02:48:19.630503+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d2adfb2e-8c08-4578-87f3-e8237f28394b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":3474823585},{"type":"v1","addr":"172.21.15.57:6801","nonce":3474823585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3474823585},{"type":"v1","addr":"172.21.15.57:6803","nonce":3474823585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3474823585},{"type":"v1","addr":"172.21.15.57:6807","nonce":3474823585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3474823585},{"type":"v1","addr":"172.21.15.57:6805","nonce":3474823585}]},"public_addr":"172.21.15.57:6801/3474823585","cluster_addr":"172.21.15.57:6803/3474823585","heartbeat_back_addr":"172.21.15.57:6807/3474823585","heartbeat_front_addr":"172.21.15.57:6805/3474823585","state":["exists","up"]},{"osd":1,"uuid":"214ac73d-ce6a-458f-a609-23472e3dbc90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3625761898},{"type":"v1","addr":"172.21.15.57:6810","nonce":3625761898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3625761898},{"type":"v1","addr":"172.21.15.57:6814","nonce":3625761898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6819","nonce":3625761898},{"type":"v1","addr":"172.21.15.57:6821","nonce":3625761898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3625761898},{"type":"v1","addr":"172.21.15.57:6818","nonce":3625761898}]},"public_addr":"172.21.15.57:6810/3625761898","cluster_addr":"172.21.15.57:6814/3625761898","heartbeat_back_addr":"172.21.15.57:6821/3625761898","heartbeat_front_addr":"172.21.15.57:6818/3625761898","state":["exists","up"]},{"osd":2,"uuid":"3bd602b2-3d2b-4b4a-9b27-598bf9731a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6809","nonce":467796665},{"type":"v1","addr":"172.21.15.57:6811","nonce":467796665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6813","nonce":467796665},{"type":"v1","addr":"172.21.15.57:6815","nonce":467796665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":467796665},{"type":"v1","addr":"172.21.15.57:6823","nonce":467796665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6817","nonce":467796665},{"type":"v1","addr":"172.21.15.57:6820","nonce":467796665}]},"public_addr":"172.21.15.57:6811/467796665","cluster_addr":"172.21.15.57:6815/467796665","heartbeat_back_addr":"172.21.15.57:6823/467796665","heartbeat_front_addr":"172.21.15.57:6820/467796665","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T02:48:10.375797+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T02:48:10.298688+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T02:48:10.355416+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-02T02:48:22.049 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2023-12-02T02:48:22.049 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-02T02:48:22.341 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:48:22.341 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":14,"fsid":"144563bd-d77d-446f-abf1-486ef977471e","created":"2023-12-02T02:48:04.334855+0000","modified":"2023-12-02T02:48:21.354734+0000","last_up_change":"2023-12-02T02:48:14.161811+0000","last_in_change":"2023-12-02T02:48:05.253733+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-02T02:48:15.878427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-02T02:48:19.630503+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d2adfb2e-8c08-4578-87f3-e8237f28394b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":3474823585},{"type":"v1","addr":"172.21.15.57:6801","nonce":3474823585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3474823585},{"type":"v1","addr":"172.21.15.57:6803","nonce":3474823585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3474823585},{"type":"v1","addr":"172.21.15.57:6807","nonce":3474823585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3474823585},{"type":"v1","addr":"172.21.15.57:6805","nonce":3474823585}]},"public_addr":"172.21.15.57:6801/3474823585","cluster_addr":"172.21.15.57:6803/3474823585","heartbeat_back_addr":"172.21.15.57:6807/3474823585","heartbeat_front_addr":"172.21.15.57:6805/3474823585","state":["exists","up"]},{"osd":1,"uuid":"214ac73d-ce6a-458f-a609-23472e3dbc90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3625761898},{"type":"v1","addr":"172.21.15.57:6810","nonce":3625761898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3625761898},{"type":"v1","addr":"172.21.15.57:6814","nonce":3625761898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6819","nonce":3625761898},{"type":"v1","addr":"172.21.15.57:6821","nonce":3625761898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3625761898},{"type":"v1","addr":"172.21.15.57:6818","nonce":3625761898}]},"public_addr":"172.21.15.57:6810/3625761898","cluster_addr":"172.21.15.57:6814/3625761898","heartbeat_back_addr":"172.21.15.57:6821/3625761898","heartbeat_front_addr":"172.21.15.57:6818/3625761898","state":["exists","up"]},{"osd":2,"uuid":"3bd602b2-3d2b-4b4a-9b27-598bf9731a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6809","nonce":467796665},{"type":"v1","addr":"172.21.15.57:6811","nonce":467796665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6813","nonce":467796665},{"type":"v1","addr":"172.21.15.57:6815","nonce":467796665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":467796665},{"type":"v1","addr":"172.21.15.57:6823","nonce":467796665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6817","nonce":467796665},{"type":"v1","addr":"172.21.15.57:6820","nonce":467796665}]},"public_addr":"172.21.15.57:6811/467796665","cluster_addr":"172.21.15.57:6815/467796665","heartbeat_back_addr":"172.21.15.57:6823/467796665","heartbeat_front_addr":"172.21.15.57:6820/467796665","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T02:48:10.375797+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T02:48:10.298688+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-02T02:48:10.355416+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-02T02:48:22.354 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2023-12-02T02:48:22.354 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2023-12-02T02:48:22.355 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2023-12-02T02:48:22.503 INFO:teuthology.orchestra.run.smithi057.stdout:34359738374 2023-12-02T02:48:22.503 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-02T02:48:22.504 INFO:teuthology.orchestra.run.smithi057.stdout:34359738372 2023-12-02T02:48:22.504 INFO:teuthology.orchestra.run.smithi057.stdout:34359738374 2023-12-02T02:48:22.505 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-12-02T02:48:22.505 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-12-02T02:48:22.847 INFO:teuthology.orchestra.run.smithi057.stdout:34359738373 2023-12-02T02:48:22.859 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.0 2023-12-02T02:48:22.871 INFO:teuthology.orchestra.run.smithi057.stdout:34359738371 2023-12-02T02:48:22.875 INFO:teuthology.orchestra.run.smithi057.stdout:34359738373 2023-12-02T02:48:22.885 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738371 for osd.1 2023-12-02T02:48:22.888 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.2 2023-12-02T02:48:23.861 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-02T02:48:23.886 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-12-02T02:48:23.889 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-12-02T02:48:24.178 INFO:teuthology.orchestra.run.smithi057.stdout:34359738373 2023-12-02T02:48:24.191 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.0 2023-12-02T02:48:24.214 INFO:teuthology.orchestra.run.smithi057.stdout:34359738373 2023-12-02T02:48:24.218 INFO:teuthology.orchestra.run.smithi057.stdout:34359738371 2023-12-02T02:48:24.227 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.2 2023-12-02T02:48:24.231 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738371 for osd.1 2023-12-02T02:48:25.192 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-02T02:48:25.227 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-12-02T02:48:25.232 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-12-02T02:48:25.514 INFO:teuthology.orchestra.run.smithi057.stdout:34359738374 2023-12-02T02:48:25.527 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.0 2023-12-02T02:48:25.527 DEBUG:teuthology.parallel:result is None 2023-12-02T02:48:25.548 INFO:teuthology.orchestra.run.smithi057.stdout:34359738374 2023-12-02T02:48:25.557 INFO:teuthology.orchestra.run.smithi057.stdout:34359738372 2023-12-02T02:48:25.561 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.2 2023-12-02T02:48:25.561 DEBUG:teuthology.parallel:result is None 2023-12-02T02:48:25.570 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738372 for osd.1 2023-12-02T02:48:25.570 DEBUG:teuthology.parallel:result is None 2023-12-02T02:48:25.571 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2023-12-02T02:48:25.571 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T02:48:25.901 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:48:25.902 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-12-02T02:48:25.913 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2023-12-02T02:48:25.825623+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":95272,"kb_used_data":1536,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283020248,"statfs":{"total":289910292480,"available":289812733952,"internally_reserved":0,"allocated":1572864,"data_stored":1288888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001140"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-02T02:48:23.846569+0000","last_change":"2023-12-02T02:48:21.369189+0000","last_active":"2023-12-02T02:48:23.846569+0000","last_peered":"2023-12-02T02:48:23.846569+0000","last_clean":"2023-12-02T02:48:23.846569+0000","last_became_active":"2023-12-02T02:48:21.367790+0000","last_became_peered":"2023-12-02T02:48:21.367790+0000","last_unstale":"2023-12-02T02:48:23.846569+0000","last_undegraded":"2023-12-02T02:48:23.846569+0000","last_fullsized":"2023-12-02T02:48:23.846569+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_clean_scrub_stamp":"2023-12-02T02:48:20.352096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T03:06:16.737870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-02T02:48:23.846587+0000","last_change":"2023-12-02T02:48:21.368993+0000","last_active":"2023-12-02T02:48:23.846587+0000","last_peered":"2023-12-02T02:48:23.846587+0000","last_clean":"2023-12-02T02:48:23.846587+0000","last_became_active":"2023-12-02T02:48:21.368048+0000","last_became_peered":"2023-12-02T02:48:21.368048+0000","last_unstale":"2023-12-02T02:48:23.846587+0000","last_undegraded":"2023-12-02T02:48:23.846587+0000","last_fullsized":"2023-12-02T02:48:23.846587+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_clean_scrub_stamp":"2023-12-02T02:48:20.352096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T03:28:15.345906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-02T02:48:23.846499+0000","last_change":"2023-12-02T02:48:21.367518+0000","last_active":"2023-12-02T02:48:23.846499+0000","last_peered":"2023-12-02T02:48:23.846499+0000","last_clean":"2023-12-02T02:48:23.846499+0000","last_became_active":"2023-12-02T02:48:21.367196+0000","last_became_peered":"2023-12-02T02:48:21.367196+0000","last_unstale":"2023-12-02T02:48:23.846499+0000","last_undegraded":"2023-12-02T02:48:23.846499+0000","last_fullsized":"2023-12-02T02:48:23.846499+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_clean_scrub_stamp":"2023-12-02T02:48:20.352096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T05:31:00.983890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-02T02:48:23.846502+0000","last_change":"2023-12-02T02:48:21.367651+0000","last_active":"2023-12-02T02:48:23.846502+0000","last_peered":"2023-12-02T02:48:23.846502+0000","last_clean":"2023-12-02T02:48:23.846502+0000","last_became_active":"2023-12-02T02:48:21.367354+0000","last_became_peered":"2023-12-02T02:48:21.367354+0000","last_unstale":"2023-12-02T02:48:23.846502+0000","last_undegraded":"2023-12-02T02:48:23.846502+0000","last_fullsized":"2023-12-02T02:48:23.846502+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_clean_scrub_stamp":"2023-12-02T02:48:20.352096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T08:08:01.749332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-02T02:48:22.887535+0000","last_change":"2023-12-02T02:48:21.367369+0000","last_active":"2023-12-02T02:48:22.887535+0000","last_peered":"2023-12-02T02:48:22.887535+0000","last_clean":"2023-12-02T02:48:22.887535+0000","last_became_active":"2023-12-02T02:48:21.367068+0000","last_became_peered":"2023-12-02T02:48:21.367068+0000","last_unstale":"2023-12-02T02:48:22.887535+0000","last_undegraded":"2023-12-02T02:48:22.887535+0000","last_fullsized":"2023-12-02T02:48:22.887535+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_clean_scrub_stamp":"2023-12-02T02:48:20.352096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T07:50:03.370248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-02T02:48:22.368189+0000","last_change":"2023-12-02T02:48:21.368234+0000","last_active":"2023-12-02T02:48:22.368189+0000","last_peered":"2023-12-02T02:48:22.368189+0000","last_clean":"2023-12-02T02:48:22.368189+0000","last_became_active":"2023-12-02T02:48:21.367848+0000","last_became_peered":"2023-12-02T02:48:21.367848+0000","last_unstale":"2023-12-02T02:48:22.368189+0000","last_undegraded":"2023-12-02T02:48:22.368189+0000","last_fullsized":"2023-12-02T02:48:22.368189+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_clean_scrub_stamp":"2023-12-02T02:48:20.352096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T12:46:09.253414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-02T02:48:22.368193+0000","last_change":"2023-12-02T02:48:21.367704+0000","last_active":"2023-12-02T02:48:22.368193+0000","last_peered":"2023-12-02T02:48:22.368193+0000","last_clean":"2023-12-02T02:48:22.368193+0000","last_became_active":"2023-12-02T02:48:21.367320+0000","last_became_peered":"2023-12-02T02:48:21.367320+0000","last_unstale":"2023-12-02T02:48:22.368193+0000","last_undegraded":"2023-12-02T02:48:22.368193+0000","last_fullsized":"2023-12-02T02:48:22.368193+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_clean_scrub_stamp":"2023-12-02T02:48:20.352096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T08:37:16.132816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-02T02:48:23.846593+0000","last_change":"2023-12-02T02:48:21.367499+0000","last_active":"2023-12-02T02:48:23.846593+0000","last_peered":"2023-12-02T02:48:23.846593+0000","last_clean":"2023-12-02T02:48:23.846593+0000","last_became_active":"2023-12-02T02:48:21.367163+0000","last_became_peered":"2023-12-02T02:48:21.367163+0000","last_unstale":"2023-12-02T02:48:23.846593+0000","last_undegraded":"2023-12-02T02:48:23.846593+0000","last_fullsized":"2023-12-02T02:48:23.846593+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_clean_scrub_stamp":"2023-12-02T02:48:20.352096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T14:29:55.084343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":141,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T02:48:21.363602+0000","last_change":"2023-12-02T02:48:17.590074+0000","last_active":"2023-12-02T02:48:21.363602+0000","last_peered":"2023-12-02T02:48:21.363602+0000","last_clean":"2023-12-02T02:48:21.363602+0000","last_became_active":"2023-12-02T02:48:17.588954+0000","last_became_peered":"2023-12-02T02:48:17.588954+0000","last_unstale":"2023-12-02T02:48:21.363602+0000","last_undegraded":"2023-12-02T02:48:21.363602+0000","last_fullsized":"2023-12-02T02:48:21.363602+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:48:16.333577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:48:16.333577+0000","last_clean_scrub_stamp":"2023-12-02T02:48:16.333577+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T09:12:18.530291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31308,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340532,"statfs":{"total":96636764160,"available":96604704768,"internally_reserved":0,"allocated":131072,"data_stored":37210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32076,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339764,"statfs":{"total":96636764160,"available":96603918336,"internally_reserved":0,"allocated":716800,"data_stored":624100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":724992,"data_stored":627578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T02:48:25.913 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T02:48:26.205 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:48:26.205 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-12-02T02:48:26.216 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2023-12-02T02:48:25.825623+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":95272,"kb_used_data":1536,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283020248,"statfs":{"total":289910292480,"available":289812733952,"internally_reserved":0,"allocated":1572864,"data_stored":1288888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001140"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-02T02:48:23.846569+0000","last_change":"2023-12-02T02:48:21.369189+0000","last_active":"2023-12-02T02:48:23.846569+0000","last_peered":"2023-12-02T02:48:23.846569+0000","last_clean":"2023-12-02T02:48:23.846569+0000","last_became_active":"2023-12-02T02:48:21.367790+0000","last_became_peered":"2023-12-02T02:48:21.367790+0000","last_unstale":"2023-12-02T02:48:23.846569+0000","last_undegraded":"2023-12-02T02:48:23.846569+0000","last_fullsized":"2023-12-02T02:48:23.846569+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_clean_scrub_stamp":"2023-12-02T02:48:20.352096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T03:06:16.737870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-02T02:48:23.846587+0000","last_change":"2023-12-02T02:48:21.368993+0000","last_active":"2023-12-02T02:48:23.846587+0000","last_peered":"2023-12-02T02:48:23.846587+0000","last_clean":"2023-12-02T02:48:23.846587+0000","last_became_active":"2023-12-02T02:48:21.368048+0000","last_became_peered":"2023-12-02T02:48:21.368048+0000","last_unstale":"2023-12-02T02:48:23.846587+0000","last_undegraded":"2023-12-02T02:48:23.846587+0000","last_fullsized":"2023-12-02T02:48:23.846587+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_clean_scrub_stamp":"2023-12-02T02:48:20.352096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T03:28:15.345906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-02T02:48:23.846499+0000","last_change":"2023-12-02T02:48:21.367518+0000","last_active":"2023-12-02T02:48:23.846499+0000","last_peered":"2023-12-02T02:48:23.846499+0000","last_clean":"2023-12-02T02:48:23.846499+0000","last_became_active":"2023-12-02T02:48:21.367196+0000","last_became_peered":"2023-12-02T02:48:21.367196+0000","last_unstale":"2023-12-02T02:48:23.846499+0000","last_undegraded":"2023-12-02T02:48:23.846499+0000","last_fullsized":"2023-12-02T02:48:23.846499+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_clean_scrub_stamp":"2023-12-02T02:48:20.352096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T05:31:00.983890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-02T02:48:23.846502+0000","last_change":"2023-12-02T02:48:21.367651+0000","last_active":"2023-12-02T02:48:23.846502+0000","last_peered":"2023-12-02T02:48:23.846502+0000","last_clean":"2023-12-02T02:48:23.846502+0000","last_became_active":"2023-12-02T02:48:21.367354+0000","last_became_peered":"2023-12-02T02:48:21.367354+0000","last_unstale":"2023-12-02T02:48:23.846502+0000","last_undegraded":"2023-12-02T02:48:23.846502+0000","last_fullsized":"2023-12-02T02:48:23.846502+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_clean_scrub_stamp":"2023-12-02T02:48:20.352096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T08:08:01.749332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-02T02:48:22.887535+0000","last_change":"2023-12-02T02:48:21.367369+0000","last_active":"2023-12-02T02:48:22.887535+0000","last_peered":"2023-12-02T02:48:22.887535+0000","last_clean":"2023-12-02T02:48:22.887535+0000","last_became_active":"2023-12-02T02:48:21.367068+0000","last_became_peered":"2023-12-02T02:48:21.367068+0000","last_unstale":"2023-12-02T02:48:22.887535+0000","last_undegraded":"2023-12-02T02:48:22.887535+0000","last_fullsized":"2023-12-02T02:48:22.887535+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_clean_scrub_stamp":"2023-12-02T02:48:20.352096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T07:50:03.370248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-02T02:48:22.368189+0000","last_change":"2023-12-02T02:48:21.368234+0000","last_active":"2023-12-02T02:48:22.368189+0000","last_peered":"2023-12-02T02:48:22.368189+0000","last_clean":"2023-12-02T02:48:22.368189+0000","last_became_active":"2023-12-02T02:48:21.367848+0000","last_became_peered":"2023-12-02T02:48:21.367848+0000","last_unstale":"2023-12-02T02:48:22.368189+0000","last_undegraded":"2023-12-02T02:48:22.368189+0000","last_fullsized":"2023-12-02T02:48:22.368189+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_clean_scrub_stamp":"2023-12-02T02:48:20.352096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T12:46:09.253414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-02T02:48:22.368193+0000","last_change":"2023-12-02T02:48:21.367704+0000","last_active":"2023-12-02T02:48:22.368193+0000","last_peered":"2023-12-02T02:48:22.368193+0000","last_clean":"2023-12-02T02:48:22.368193+0000","last_became_active":"2023-12-02T02:48:21.367320+0000","last_became_peered":"2023-12-02T02:48:21.367320+0000","last_unstale":"2023-12-02T02:48:22.368193+0000","last_undegraded":"2023-12-02T02:48:22.368193+0000","last_fullsized":"2023-12-02T02:48:22.368193+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_clean_scrub_stamp":"2023-12-02T02:48:20.352096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T08:37:16.132816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-02T02:48:23.846593+0000","last_change":"2023-12-02T02:48:21.367499+0000","last_active":"2023-12-02T02:48:23.846593+0000","last_peered":"2023-12-02T02:48:23.846593+0000","last_clean":"2023-12-02T02:48:23.846593+0000","last_became_active":"2023-12-02T02:48:21.367163+0000","last_became_peered":"2023-12-02T02:48:21.367163+0000","last_unstale":"2023-12-02T02:48:23.846593+0000","last_undegraded":"2023-12-02T02:48:23.846593+0000","last_fullsized":"2023-12-02T02:48:23.846593+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:48:20.352096+0000","last_clean_scrub_stamp":"2023-12-02T02:48:20.352096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T14:29:55.084343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":141,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-02T02:48:21.363602+0000","last_change":"2023-12-02T02:48:17.590074+0000","last_active":"2023-12-02T02:48:21.363602+0000","last_peered":"2023-12-02T02:48:21.363602+0000","last_clean":"2023-12-02T02:48:21.363602+0000","last_became_active":"2023-12-02T02:48:17.588954+0000","last_became_peered":"2023-12-02T02:48:17.588954+0000","last_unstale":"2023-12-02T02:48:21.363602+0000","last_undegraded":"2023-12-02T02:48:21.363602+0000","last_fullsized":"2023-12-02T02:48:21.363602+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-02T02:48:16.333577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-02T02:48:16.333577+0000","last_clean_scrub_stamp":"2023-12-02T02:48:16.333577+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T09:12:18.530291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31308,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340532,"statfs":{"total":96636764160,"available":96604704768,"internally_reserved":0,"allocated":131072,"data_stored":37210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32076,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339764,"statfs":{"total":96636764160,"available":96603918336,"internally_reserved":0,"allocated":716800,"data_stored":624100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":724992,"data_stored":627578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-02T02:48:26.217 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2023-12-02T02:48:26.217 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2023-12-02T02:48:26.217 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2023-12-02T02:48:26.218 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2023-12-02T02:48:26.545 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-12-02T02:48:26.545 INFO:teuthology.orchestra.run.smithi057.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-02T02:48:26.557 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2023-12-02T02:48:26.558 INFO:teuthology.run_tasks:Running task exec... 2023-12-02T02:48:26.570 INFO:teuthology.task.exec:Executing custom commands... 2023-12-02T02:48:26.570 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi057.front.sepia.ceph.com 2023-12-02T02:48:26.570 DEBUG:teuthology.orchestra.run.smithi057:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-12-02T02:48:26.947 INFO:teuthology.orchestra.run.smithi057.stdout:[client.mirror] 2023-12-02T02:48:26.948 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQCwmmpltZfGNxAAUDELwC8t5wEVpF6awg2ziw== 2023-12-02T02:48:26.948 INFO:teuthology.orchestra.run.smithi057.stdout: caps mon = "profile rbd-mirror-peer" 2023-12-02T02:48:26.948 INFO:teuthology.orchestra.run.smithi057.stdout: caps osd = "profile rbd" 2023-12-02T02:48:26.948 INFO:teuthology.orchestra.run.smithi057.stderr:updated caps for client.mirror 2023-12-02T02:48:26.960 DEBUG:teuthology.orchestra.run.smithi057:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-02T02:48:27.342 INFO:teuthology.orchestra.run.smithi057.stdout:[client.mirror.0] 2023-12-02T02:48:27.342 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQCxmmpldJwCABAAaY+tg5WkEoZozXkdq6PvVA== 2023-12-02T02:48:27.342 INFO:teuthology.orchestra.run.smithi057.stdout: caps mon = "profile rbd-mirror" 2023-12-02T02:48:27.342 INFO:teuthology.orchestra.run.smithi057.stdout: caps osd = "profile rbd" 2023-12-02T02:48:27.343 INFO:teuthology.orchestra.run.smithi057.stderr:updated caps for client.mirror.0 2023-12-02T02:48:27.356 DEBUG:teuthology.orchestra.run.smithi057:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-02T02:48:27.734 INFO:teuthology.orchestra.run.smithi057.stdout:[client.mirror.1] 2023-12-02T02:48:27.734 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQCxmmpl/KzZAxAAAHNU5BxNOS4Hb7mlP8xfiQ== 2023-12-02T02:48:27.734 INFO:teuthology.orchestra.run.smithi057.stdout: caps mon = "profile rbd-mirror" 2023-12-02T02:48:27.735 INFO:teuthology.orchestra.run.smithi057.stdout: caps osd = "profile rbd" 2023-12-02T02:48:27.735 INFO:teuthology.orchestra.run.smithi057.stderr:updated caps for client.mirror.1 2023-12-02T02:48:27.747 DEBUG:teuthology.orchestra.run.smithi057:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-02T02:48:28.126 INFO:teuthology.orchestra.run.smithi057.stdout:[client.mirror.2] 2023-12-02T02:48:28.127 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQCxmmplYLqpBxAAyX536ockZP9zBJos7q7wtA== 2023-12-02T02:48:28.127 INFO:teuthology.orchestra.run.smithi057.stdout: caps mon = "profile rbd-mirror" 2023-12-02T02:48:28.127 INFO:teuthology.orchestra.run.smithi057.stdout: caps osd = "profile rbd" 2023-12-02T02:48:28.127 INFO:teuthology.orchestra.run.smithi057.stderr:updated caps for client.mirror.2 2023-12-02T02:48:28.140 DEBUG:teuthology.orchestra.run.smithi057:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-02T02:48:28.519 INFO:teuthology.orchestra.run.smithi057.stdout:[client.mirror.3] 2023-12-02T02:48:28.519 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQCxmmplDH9+CxAA7e5cNOlSchPQ2fLTpCCDQA== 2023-12-02T02:48:28.519 INFO:teuthology.orchestra.run.smithi057.stdout: caps mon = "profile rbd-mirror" 2023-12-02T02:48:28.519 INFO:teuthology.orchestra.run.smithi057.stdout: caps osd = "profile rbd" 2023-12-02T02:48:28.520 INFO:teuthology.orchestra.run.smithi057.stderr:updated caps for client.mirror.3 2023-12-02T02:48:28.532 DEBUG:teuthology.orchestra.run.smithi057:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-12-02T02:48:28.908 INFO:teuthology.orchestra.run.smithi057.stdout:[client.mirror] 2023-12-02T02:48:28.908 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDUmmplOzQ6CRAAUIjVJQJPSGS1GzrsKTYsbw== 2023-12-02T02:48:28.909 INFO:teuthology.orchestra.run.smithi057.stdout: caps mon = "profile rbd-mirror-peer" 2023-12-02T02:48:28.909 INFO:teuthology.orchestra.run.smithi057.stdout: caps osd = "profile rbd" 2023-12-02T02:48:28.909 INFO:teuthology.orchestra.run.smithi057.stderr:updated caps for client.mirror 2023-12-02T02:48:28.922 DEBUG:teuthology.orchestra.run.smithi057:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-02T02:48:29.304 INFO:teuthology.orchestra.run.smithi057.stdout:[client.mirror.0] 2023-12-02T02:48:29.304 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDUmmplt/0JDRAAk9zJSwtvwxUyREdQ2ykhDA== 2023-12-02T02:48:29.304 INFO:teuthology.orchestra.run.smithi057.stdout: caps mon = "profile rbd-mirror" 2023-12-02T02:48:29.304 INFO:teuthology.orchestra.run.smithi057.stdout: caps osd = "profile rbd" 2023-12-02T02:48:29.304 INFO:teuthology.orchestra.run.smithi057.stderr:updated caps for client.mirror.0 2023-12-02T02:48:29.318 DEBUG:teuthology.orchestra.run.smithi057:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-02T02:48:29.699 INFO:teuthology.orchestra.run.smithi057.stdout:[client.mirror.1] 2023-12-02T02:48:29.699 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDUmmpl6uDiEBAAJraUZqpIoEk51eDpAxreUA== 2023-12-02T02:48:29.699 INFO:teuthology.orchestra.run.smithi057.stdout: caps mon = "profile rbd-mirror" 2023-12-02T02:48:29.699 INFO:teuthology.orchestra.run.smithi057.stdout: caps osd = "profile rbd" 2023-12-02T02:48:29.699 INFO:teuthology.orchestra.run.smithi057.stderr:updated caps for client.mirror.1 2023-12-02T02:48:29.712 DEBUG:teuthology.orchestra.run.smithi057:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-02T02:48:30.090 INFO:teuthology.orchestra.run.smithi057.stdout:[client.mirror.2] 2023-12-02T02:48:30.090 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDUmmplpo2xFBAAdVgxqWwmj4YzVeeCsJheGA== 2023-12-02T02:48:30.090 INFO:teuthology.orchestra.run.smithi057.stdout: caps mon = "profile rbd-mirror" 2023-12-02T02:48:30.090 INFO:teuthology.orchestra.run.smithi057.stdout: caps osd = "profile rbd" 2023-12-02T02:48:30.090 INFO:teuthology.orchestra.run.smithi057.stderr:updated caps for client.mirror.2 2023-12-02T02:48:30.103 DEBUG:teuthology.orchestra.run.smithi057:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-02T02:48:30.488 INFO:teuthology.orchestra.run.smithi057.stdout:[client.mirror.3] 2023-12-02T02:48:30.488 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDUmmplFwGHGBAAVCmykGSiXSCrVxCIpPAzjQ== 2023-12-02T02:48:30.488 INFO:teuthology.orchestra.run.smithi057.stdout: caps mon = "profile rbd-mirror" 2023-12-02T02:48:30.489 INFO:teuthology.orchestra.run.smithi057.stdout: caps osd = "profile rbd" 2023-12-02T02:48:30.489 INFO:teuthology.orchestra.run.smithi057.stderr:updated caps for client.mirror.3 2023-12-02T02:48:30.501 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-02T02:48:30.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-02T02:48:30.526 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-02T02:48:30.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-02T02:48:30.529 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-02T02:48:30.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-02T02:48:30.540 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-02T02:48:30.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-02T02:48:30.543 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-02T02:48:30.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-02T02:48:30.573 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-02T02:48:30.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-02T02:48:30.575 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-02T02:48:30.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-02T02:48:30.605 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-02T02:48:30.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-02T02:48:30.608 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-02T02:48:30.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-02T02:48:30.640 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-02T02:48:30.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-02T02:48:30.642 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-02T02:48:30.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-02T02:48:30.673 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-02T02:48:30.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-02T02:48:30.675 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-02T02:48:30.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-02T02:48:30.710 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-02T02:48:30.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-02T02:48:30.712 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-02T02:48:30.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-12-02T02:48:30.771 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-12-02T02:48:30.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-12-02T02:48:30.775 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-12-02T02:48:30.810 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1701485310 2023-12-02T02:48:30.811 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-12-02T02:48:30.811 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-12-02T02:48:30.841 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1701485310 2023-12-02T02:48:30.841 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-12-02T02:48:30.841 INFO:teuthology.run_tasks:Running task workunit... 2023-12-02T02:48:30.879 INFO:tasks.workunit:Pulling workunits from ref bed7681480ccb0192b91a35e691658867e1e96b7 2023-12-02T02:48:30.880 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-02T02:48:30.880 DEBUG:teuthology.orchestra.run.smithi057:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-02T02:48:30.880 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2023-12-02T02:48:30.880 INFO:tasks.rbd_mirror_thrash:waiting for 88 secs before thrashing 2023-12-02T02:48:30.881 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2023-12-02T02:48:30.881 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2023-12-02T02:48:30.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-02T02:48:30.895 INFO:teuthology.orchestra.run.smithi057.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2023-12-02T02:48:30.896 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-02T02:48:30.952 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-02T02:48:30.954 DEBUG:teuthology.orchestra.run.smithi057:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2023-12-02T02:48:31.007 INFO:tasks.workunit:timeout=6h 2023-12-02T02:48:31.007 INFO:tasks.workunit:cleanup=True 2023-12-02T02:48:31.007 DEBUG:teuthology.orchestra.run.smithi057:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout bed7681480ccb0192b91a35e691658867e1e96b7 2023-12-02T02:48:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2023-12-02T02:49:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Note: switching to 'bed7681480ccb0192b91a35e691658867e1e96b7'. 2023-12-02T02:49:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:49:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-02T02:49:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-02T02:49:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:state without impacting any branches by switching back to a branch. 2023-12-02T02:49:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:49:18.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-02T02:49:18.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-02T02:49:18.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:49:18.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: git switch -c 2023-12-02T02:49:18.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:49:18.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Or undo this operation with: 2023-12-02T02:49:18.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:49:18.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: git switch - 2023-12-02T02:49:18.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:49:18.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-02T02:49:18.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:49:18.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:HEAD is now at bed7681480c Merge pull request #54648 from rhcs-dashboard/fs-rename-fix 2023-12-02T02:49:18.928 DEBUG:teuthology.orchestra.run.smithi057:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2023-12-02T02:49:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-02T02:49:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-12-02T02:49:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-02T02:49:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-02T02:49:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-02T02:49:19.884 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-12-02T02:49:19.886 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-12-02T02:49:19.886 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-02T02:49:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-12-02T02:49:19.929 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-12-02T02:49:19.930 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2023-12-02T02:49:19.985 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2023-12-02T02:49:19.987 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2023-12-02T02:49:19.987 DEBUG:teuthology.orchestra.run.smithi057:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bed7681480ccb0192b91a35e691658867e1e96b7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-02T02:49:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ IMAGE_COUNT=50 2023-12-02T02:49:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ export LOCKDEP=0 2023-12-02T02:49:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ LOCKDEP=0 2023-12-02T02:49:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-02T02:49:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2023-12-02T02:49:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ type xmlstarlet 2023-12-02T02:49:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ XMLSTARLET=xmlstarlet 2023-12-02T02:49:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ RBD_MIRROR_INSTANCES=4 2023-12-02T02:49:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ CLUSTER1=cluster1 2023-12-02T02:49:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ CLUSTER2=cluster2 2023-12-02T02:49:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ PEER_CLUSTER_SUFFIX= 2023-12-02T02:49:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ POOL=mirror 2023-12-02T02:49:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ PARENT_POOL=mirror_parent 2023-12-02T02:49:20.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ NS1=ns1 2023-12-02T02:49:20.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ NS2=ns2 2023-12-02T02:49:20.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ TEMPDIR= 2023-12-02T02:49:20.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ CEPH_ID=mirror 2023-12-02T02:49:20.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ RBD_IMAGE_FEATURES=layering 2023-12-02T02:49:20.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2023-12-02T02:49:20.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ RBD_MIRROR_MODE=snapshot 2023-12-02T02:49:20.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ MIRROR_POOL_MODE=image 2023-12-02T02:49:20.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' snapshot = snapshot ']' 2023-12-02T02:49:20.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ MIRROR_POOL_MODE=image 2023-12-02T02:49:20.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ export 'CEPH_ARGS=--id mirror' 2023-12-02T02:49:20.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ CEPH_ARGS='--id mirror' 2023-12-02T02:49:20.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ LAST_MIRROR_INSTANCE=3 2023-12-02T02:49:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-02T02:49:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2023-12-02T02:49:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2023-12-02T02:49:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ CEPH_BIN=. 2023-12-02T02:49:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ CEPH_SRC=. 2023-12-02T02:49:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' -e CMakeCache.txt ']' 2023-12-02T02:49:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 -gt 0 ']' 2023-12-02T02:49:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ setup 2023-12-02T02:49:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local c 2023-12-02T02:49:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ trap 'cleanup $?' INT TERM EXIT 2023-12-02T02:49:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ setup_tempdir 2023-12-02T02:49:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T02:49:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mktemp -d 2023-12-02T02:49:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ TEMPDIR=/tmp/tmp.kebs98RNzE 2023-12-02T02:49:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -z 1 ']' 2023-12-02T02:49:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ setup_pools cluster1 cluster2 2023-12-02T02:49:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:49:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local remote_cluster=cluster2 2023-12-02T02:49:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mon_map_file 2023-12-02T02:49:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mon_addr 2023-12-02T02:49:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local admin_key_file 2023-12-02T02:49:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local uuid 2023-12-02T02:49:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:49:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2023-12-02T02:49:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:pool 'mirror' already exists 2023-12-02T02:49:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:49:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2023-12-02T02:49:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:pool 'mirror_parent' already exists 2023-12-02T02:49:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:49:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 pool init mirror 2023-12-02T02:49:25.331 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:49:25.331 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2023-12-02T02:49:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T02:49:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:49:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2023-12-02T02:49:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2023-12-02T02:49:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2023-12-02T02:49:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2023-12-02T02:49:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2023-12-02T02:49:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2023-12-02T02:49:28.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -z ']' 2023-12-02T02:49:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -z ']' 2023-12-02T02:49:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ peer_add cluster1 mirror cluster2 2023-12-02T02:49:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:49:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:49:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:49:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:49:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local client_cluster=cluster2 2023-12-02T02:49:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:49:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local remote_cluster=cluster2 2023-12-02T02:49:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local uuid_var_name 2023-12-02T02:49:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T02:49:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local error_code 2023-12-02T02:49:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local peer_uuid 2023-12-02T02:49:28.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 16 32 2023-12-02T02:49:28.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T02:49:28.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2023-12-02T02:49:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ peer_uuid=06fbe0df-d787-44c2-aede-d6f6ace95f83 2023-12-02T02:49:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=0 2023-12-02T02:49:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T02:49:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 0 -eq 17 ']' 2023-12-02T02:49:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 0 -eq 0 2023-12-02T02:49:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T02:49:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ peer_add cluster1 mirror_parent cluster2 2023-12-02T02:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror_parent 2023-12-02T02:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local client_cluster=cluster2 2023-12-02T02:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local remote_cluster=cluster2 2023-12-02T02:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local uuid_var_name 2023-12-02T02:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T02:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local error_code 2023-12-02T02:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local peer_uuid 2023-12-02T02:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 16 32 2023-12-02T02:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T02:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2023-12-02T02:49:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ peer_uuid=23f8ff71-737e-4eab-b981-22885d6c63ed 2023-12-02T02:49:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=0 2023-12-02T02:49:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T02:49:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 0 -eq 17 ']' 2023-12-02T02:49:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 0 -eq 0 2023-12-02T02:49:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T02:49:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:49:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ setup_pools cluster2 cluster1 2023-12-02T02:49:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:49:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local remote_cluster=cluster1 2023-12-02T02:49:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mon_map_file 2023-12-02T02:49:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mon_addr 2023-12-02T02:49:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local admin_key_file 2023-12-02T02:49:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local uuid 2023-12-02T02:49:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:49:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2023-12-02T02:49:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:pool 'mirror' already exists 2023-12-02T02:49:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:49:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2023-12-02T02:49:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:pool 'mirror_parent' already exists 2023-12-02T02:49:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:49:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 pool init mirror 2023-12-02T02:49:30.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi057.stderr:2023-12-02T02:49:30.689+0000 7f800102e640 -1 rbd::mirror::PoolReplayer: 0x556841cd0c00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-02T02:49:30.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr:2023-12-02T02:49:30.697+0000 7fe3a9e0f640 -1 rbd::mirror::PoolReplayer: 0x5630e7456c00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-02T02:49:30.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi057.stderr:2023-12-02T02:49:30.726+0000 7f2964e0f640 -1 rbd::mirror::PoolReplayer: 0x561c9f100c00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-02T02:49:30.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi057.stderr:2023-12-02T02:49:30.891+0000 7f9508f6d640 -1 rbd::mirror::PoolReplayer: 0x55658766ec00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-02T02:49:31.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi057.stderr:2023-12-02T02:49:31.405+0000 7f9508f6d640 -1 rbd::mirror::PoolReplayer: 0x55658775d500 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-02T02:49:31.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr:2023-12-02T02:49:31.405+0000 7fe3a9e0f640 -1 rbd::mirror::PoolReplayer: 0x5630e75e7500 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-02T02:49:31.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi057.stderr:2023-12-02T02:49:31.543+0000 7f800102e640 -1 rbd::mirror::PoolReplayer: 0x556841e47500 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-02T02:49:31.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi057.stderr:2023-12-02T02:49:31.543+0000 7f2964e0f640 -1 rbd::mirror::PoolReplayer: 0x561c9f2cd500 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-02T02:49:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:49:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2023-12-02T02:49:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T02:49:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:49:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2023-12-02T02:49:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2023-12-02T02:49:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2023-12-02T02:49:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2023-12-02T02:49:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2023-12-02T02:49:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2023-12-02T02:49:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -z ']' 2023-12-02T02:49:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -z ']' 2023-12-02T02:49:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ peer_add cluster2 mirror cluster1 2023-12-02T02:49:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:49:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:49:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:49:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:49:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local client_cluster=cluster1 2023-12-02T02:49:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:49:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local remote_cluster=cluster1 2023-12-02T02:49:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local uuid_var_name 2023-12-02T02:49:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T02:49:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local error_code 2023-12-02T02:49:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local peer_uuid 2023-12-02T02:49:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 16 32 2023-12-02T02:49:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T02:49:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2023-12-02T02:49:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ peer_uuid=dc90bdbc-2bd4-4e6e-908c-8a21270f2c3a 2023-12-02T02:49:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=0 2023-12-02T02:49:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T02:49:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 0 -eq 17 ']' 2023-12-02T02:49:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 0 -eq 0 2023-12-02T02:49:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T02:49:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:49:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ peer_add cluster2 mirror_parent cluster1 2023-12-02T02:49:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:49:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:49:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror_parent 2023-12-02T02:49:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:49:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local client_cluster=cluster1 2023-12-02T02:49:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:49:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local remote_cluster=cluster1 2023-12-02T02:49:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local uuid_var_name 2023-12-02T02:49:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T02:49:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local error_code 2023-12-02T02:49:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local peer_uuid 2023-12-02T02:49:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 16 32 2023-12-02T02:49:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T02:49:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2023-12-02T02:49:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ peer_uuid=4b0bce83-98aa-4068-b6f7-cf0a841c7451 2023-12-02T02:49:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=0 2023-12-02T02:49:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T02:49:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 0 -eq 17 ']' 2023-12-02T02:49:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 0 -eq 0 2023-12-02T02:49:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T02:49:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:49:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T02:49:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ start_mirrors cluster1 2023-12-02T02:49:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:49:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ seq 0 3 2023-12-02T02:49:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ start_mirror cluster1:0 2023-12-02T02:49:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1:0 2023-12-02T02:49:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local instance 2023-12-02T02:49:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-02T02:49:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local val=cluster1:0 2023-12-02T02:49:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster_var_name=cluster 2023-12-02T02:49:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local instance_var_name=instance 2023-12-02T02:49:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cluster=cluster1 2023-12-02T02:49:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ instance=0 2023-12-02T02:49:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 0 = cluster1:0 ']' 2023-12-02T02:49:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ eval cluster=cluster1 2023-12-02T02:49:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:49:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ eval instance=0 2023-12-02T02:49:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T02:49:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test -n 1 2023-12-02T02:49:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 2023-12-02T02:49:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ start_mirror cluster1:1 2023-12-02T02:49:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1:1 2023-12-02T02:49:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local instance 2023-12-02T02:49:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-02T02:49:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local val=cluster1:1 2023-12-02T02:49:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster_var_name=cluster 2023-12-02T02:49:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local instance_var_name=instance 2023-12-02T02:49:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cluster=cluster1 2023-12-02T02:49:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ instance=1 2023-12-02T02:49:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 1 = cluster1:1 ']' 2023-12-02T02:49:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ eval cluster=cluster1 2023-12-02T02:49:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:49:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ eval instance=1 2023-12-02T02:49:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T02:49:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test -n 1 2023-12-02T02:49:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 2023-12-02T02:49:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ start_mirror cluster1:2 2023-12-02T02:49:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1:2 2023-12-02T02:49:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local instance 2023-12-02T02:49:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-02T02:49:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local val=cluster1:2 2023-12-02T02:49:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster_var_name=cluster 2023-12-02T02:49:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local instance_var_name=instance 2023-12-02T02:49:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cluster=cluster1 2023-12-02T02:49:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ instance=2 2023-12-02T02:49:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 2 = cluster1:2 ']' 2023-12-02T02:49:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ eval cluster=cluster1 2023-12-02T02:49:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:49:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ eval instance=2 2023-12-02T02:49:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T02:49:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test -n 1 2023-12-02T02:49:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 2023-12-02T02:49:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ start_mirror cluster1:3 2023-12-02T02:49:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1:3 2023-12-02T02:49:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local instance 2023-12-02T02:49:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-02T02:49:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local val=cluster1:3 2023-12-02T02:49:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster_var_name=cluster 2023-12-02T02:49:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local instance_var_name=instance 2023-12-02T02:49:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cluster=cluster1 2023-12-02T02:49:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ instance=3 2023-12-02T02:49:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 3 = cluster1:3 ']' 2023-12-02T02:49:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ eval cluster=cluster1 2023-12-02T02:49:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:49:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ eval instance=3 2023-12-02T02:49:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T02:49:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test -n 1 2023-12-02T02:49:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 2023-12-02T02:49:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ start_mirrors cluster2 2023-12-02T02:49:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:49:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ seq 0 3 2023-12-02T02:49:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ start_mirror cluster2:0 2023-12-02T02:49:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2:0 2023-12-02T02:49:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local instance 2023-12-02T02:49:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set_cluster_instance cluster2:0 cluster instance 2023-12-02T02:49:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local val=cluster2:0 2023-12-02T02:49:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster_var_name=cluster 2023-12-02T02:49:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local instance_var_name=instance 2023-12-02T02:49:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cluster=cluster2 2023-12-02T02:49:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ instance=0 2023-12-02T02:49:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 0 = cluster2:0 ']' 2023-12-02T02:49:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ eval cluster=cluster2 2023-12-02T02:49:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster2 2023-12-02T02:49:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ eval instance=0 2023-12-02T02:49:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T02:49:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test -n 1 2023-12-02T02:49:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 2023-12-02T02:49:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ start_mirror cluster2:1 2023-12-02T02:49:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2:1 2023-12-02T02:49:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local instance 2023-12-02T02:49:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set_cluster_instance cluster2:1 cluster instance 2023-12-02T02:49:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local val=cluster2:1 2023-12-02T02:49:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster_var_name=cluster 2023-12-02T02:49:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local instance_var_name=instance 2023-12-02T02:49:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cluster=cluster2 2023-12-02T02:49:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ instance=1 2023-12-02T02:49:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 1 = cluster2:1 ']' 2023-12-02T02:49:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ eval cluster=cluster2 2023-12-02T02:49:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster2 2023-12-02T02:49:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ eval instance=1 2023-12-02T02:49:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T02:49:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test -n 1 2023-12-02T02:49:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 2023-12-02T02:49:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ start_mirror cluster2:2 2023-12-02T02:49:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2:2 2023-12-02T02:49:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local instance 2023-12-02T02:49:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set_cluster_instance cluster2:2 cluster instance 2023-12-02T02:49:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local val=cluster2:2 2023-12-02T02:49:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster_var_name=cluster 2023-12-02T02:49:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local instance_var_name=instance 2023-12-02T02:49:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cluster=cluster2 2023-12-02T02:49:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ instance=2 2023-12-02T02:49:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 2 = cluster2:2 ']' 2023-12-02T02:49:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ eval cluster=cluster2 2023-12-02T02:49:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster2 2023-12-02T02:49:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ eval instance=2 2023-12-02T02:49:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T02:49:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test -n 1 2023-12-02T02:49:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 2023-12-02T02:49:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ start_mirror cluster2:3 2023-12-02T02:49:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2:3 2023-12-02T02:49:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local instance 2023-12-02T02:49:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set_cluster_instance cluster2:3 cluster instance 2023-12-02T02:49:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local val=cluster2:3 2023-12-02T02:49:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster_var_name=cluster 2023-12-02T02:49:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local instance_var_name=instance 2023-12-02T02:49:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cluster=cluster2 2023-12-02T02:49:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ instance=3 2023-12-02T02:49:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 3 = cluster2:3 ']' 2023-12-02T02:49:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ eval cluster=cluster2 2023-12-02T02:49:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster2 2023-12-02T02:49:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ eval instance=3 2023-12-02T02:49:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T02:49:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test -n 1 2023-12-02T02:49:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 2023-12-02T02:49:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2023-12-02T02:49:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ tee -a /tmp/tmp.kebs98RNzE/rbd-mirror.test.log 2023-12-02T02:49:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ date '+%F %T' 2023-12-02T02:49:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02 02:49:36 TEST: add image and test replay after client crashes 2023-12-02T02:49:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:2023-12-02 02:49:36 TEST: add image and test replay after client crashes 2023-12-02T02:49:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=test 2023-12-02T02:49:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2023-12-02T02:49:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:49:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:49:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:49:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:49:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:49:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:49:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T02:49:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T02:49:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:49:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror test 512M 2023-12-02T02:49:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:49:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:49:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:49:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:49:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:49:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:49:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T02:49:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 512M ']' 2023-12-02T02:49:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=512M 2023-12-02T02:49:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:49:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/test 2023-12-02T02:49:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T02:49:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:49:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T02:49:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:49:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror test snapshot 2023-12-02T02:49:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:49:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:49:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:49:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T02:49:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2023-12-02T02:49:37.491 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T02:49:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/test 2023-12-02T02:49:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'test': 2023-12-02T02:49:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 512 MiB in 128 objects 2023-12-02T02:49:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T02:49:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T02:49:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 10bdeee7c170 2023-12-02T02:49:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.10bdeee7c170 2023-12-02T02:49:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T02:49:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T02:49:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T02:49:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T02:49:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 02:49:36 2023 2023-12-02T02:49:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 02:49:36 2023 2023-12-02T02:49:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 02:49:36 2023 2023-12-02T02:49:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T02:49:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T02:49:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:49:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T02:49:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-02T02:49:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:49:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:49:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:49:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-02T02:49:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:49:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:49:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:49:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T02:49:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T02:49:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:49:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:49:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-02T02:49:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:49:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:49:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:49:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T02:49:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T02:49:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T02:49:38.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-02T02:49:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T02:49:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T02:49:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:49:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T02:49:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T02:49:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T02:49:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:49:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T02:49:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T02:49:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T02:49:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T02:49:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T02:49:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:49:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:49:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T02:49:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T02:49:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:49:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:49:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:49:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T02:49:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T02:49:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:49:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T02:49:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:49:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T02:49:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T02:49:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:49:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:49:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T02:49:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T02:49:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:49:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:49:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:49:38.549 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T02:49:38.549 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:49:38.549 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:49:38.549 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T02:49:38.549 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T02:49:38.549 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T02:49:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:49:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:49:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:49:38.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T02:49:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:49:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:49:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:49:38.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:38.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-02T02:49:38.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T02:49:38.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:49:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:49:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T02:49:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T02:49:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:49:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:49:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:49:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T02:49:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T02:49:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:49:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T02:49:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:49:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T02:49:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T02:49:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:49:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:49:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T02:49:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T02:49:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:49:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:49:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:49:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T02:49:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:49:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:49:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T02:49:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T02:49:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T02:49:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:49:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:49:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:49:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-02T02:49:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:49:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:49:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:49:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-02T02:49:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T02:49:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:49:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:49:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T02:49:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T02:49:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:49:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:49:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:49:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T02:49:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T02:49:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:49:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T02:49:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:49:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T02:49:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T02:49:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:49:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:49:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T02:49:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T02:49:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:49:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:49:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:49:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T02:49:38.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:49:38.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:49:38.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T02:49:38.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T02:49:38.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T02:49:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:49:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:49:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:49:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-02T02:49:38.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:49:38.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:49:38.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:49:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:38.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-02T02:49:38.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T02:49:38.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:49:38.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:49:38.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T02:49:38.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T02:49:38.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:49:38.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:49:38.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:49:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T02:49:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T02:49:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:49:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T02:49:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T02:49:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T02:49:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T02:49:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:49:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:49:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T02:49:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T02:49:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:49:38.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:49:38.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:38.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T02:49:38.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T02:49:38.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:49:38.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:49:38.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T02:49:38.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T02:49:38.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T02:49:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T02:49:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T02:49:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T02:49:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-02T02:49:38.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:49:38.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:49:38.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:49:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:49:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 1 2023-12-02T02:49:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T02:49:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T02:49:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T02:49:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T02:49:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:49:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:49:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T02:49:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T02:49:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:49:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:49:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:49:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T02:49:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T02:49:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:49:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T02:49:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:49:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T02:49:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T02:49:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:49:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:49:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T02:49:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T02:49:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:49:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:49:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:49:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T02:49:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:49:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:49:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T02:49:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T02:49:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T02:49:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:49:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:49:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:49:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T02:49:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:49:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:49:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:49:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-02T02:49:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T02:49:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:49:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:49:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T02:49:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T02:49:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:49:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:49:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:49:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T02:49:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T02:49:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:49:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T02:49:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:49:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T02:49:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T02:49:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:49:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:49:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T02:49:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T02:49:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:49:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:49:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:49:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T02:49:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:49:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:49:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T02:49:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T02:49:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T02:49:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:49:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:49:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:49:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-02T02:49:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:49:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:49:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:49:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-02T02:49:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T02:49:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:49:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:49:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T02:49:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T02:49:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:49:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:49:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:49:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T02:49:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T02:49:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:49:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T02:49:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:49:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T02:49:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T02:49:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:49:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:49:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T02:49:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T02:49:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:49:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:49:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:49:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T02:49:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:49:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:49:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T02:49:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T02:49:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T02:49:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:49:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:49:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:49:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-02T02:49:40.224 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:49:40.224 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:49:40.224 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:49:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-02T02:49:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T02:49:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:49:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:49:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T02:49:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T02:49:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:49:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:49:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:49:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T02:49:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T02:49:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:49:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T02:49:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T02:49:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T02:49:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T02:49:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:49:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:49:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T02:49:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T02:49:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:49:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:49:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T02:49:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T02:49:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:49:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:49:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T02:49:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T02:49:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T02:49:40.245 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T02:49:40.245 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T02:49:40.245 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T02:49:40.245 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-02T02:49:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:49:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:49:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:49:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:49:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 2 2023-12-02T02:49:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T02:49:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T02:49:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T02:49:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T02:49:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:49:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:49:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T02:49:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T02:49:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:49:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:49:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:49:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T02:49:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T02:49:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:49:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T02:49:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:49:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T02:49:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T02:49:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:49:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:49:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T02:49:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T02:49:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:49:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:49:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:49:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T02:49:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:49:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:49:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T02:49:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T02:49:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T02:49:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:49:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:49:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:49:42.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T02:49:42.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:49:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:49:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:49:42.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:42.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-02T02:49:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T02:49:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:49:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:49:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T02:49:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T02:49:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:49:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:49:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:49:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T02:49:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T02:49:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:49:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T02:49:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:49:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T02:49:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T02:49:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:49:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:49:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T02:49:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T02:49:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:49:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:49:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:49:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T02:49:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:49:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:49:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T02:49:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T02:49:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T02:49:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:49:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:49:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:49:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-02T02:49:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:49:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:49:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:49:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-02T02:49:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T02:49:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:49:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:49:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T02:49:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T02:49:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:49:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:49:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:49:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T02:49:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T02:49:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:49:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T02:49:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:49:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T02:49:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T02:49:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:49:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:49:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T02:49:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T02:49:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:49:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:49:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:49:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T02:49:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:49:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:49:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T02:49:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T02:49:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T02:49:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:49:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:49:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:49:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-02T02:49:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:49:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:49:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:49:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-02T02:49:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T02:49:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:49:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:49:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T02:49:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T02:49:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:49:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:49:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:49:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T02:49:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T02:49:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:49:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T02:49:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T02:49:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T02:49:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T02:49:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:49:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:49:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T02:49:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T02:49:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:49:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:49:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T02:49:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T02:49:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:49:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:49:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T02:49:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T02:49:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T02:49:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T02:49:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T02:49:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T02:49:42.637 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-02T02:49:42.709 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:49:42.709 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:49:42.709 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:49:42.710 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:49:42.710 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 4 2023-12-02T02:49:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T02:49:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T02:49:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T02:49:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T02:49:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:49:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:49:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T02:49:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T02:49:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:49:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:49:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:49:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T02:49:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T02:49:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:49:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T02:49:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:49:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T02:49:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T02:49:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:49:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:49:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T02:49:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T02:49:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:49:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:49:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:49:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T02:49:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:49:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:49:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T02:49:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T02:49:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T02:49:46.735 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:49:46.736 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:49:46.736 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:49:46.736 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T02:49:46.815 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:49:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:49:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:49:46.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:46.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-02T02:49:46.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T02:49:46.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:49:46.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:49:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T02:49:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T02:49:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:49:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:49:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:49:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T02:49:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T02:49:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:49:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T02:49:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:49:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T02:49:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T02:49:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:49:46.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:49:46.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T02:49:46.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T02:49:46.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:49:46.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:49:46.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:46.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:49:46.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T02:49:46.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:49:46.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:49:46.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T02:49:46.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T02:49:46.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T02:49:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:49:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:49:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:49:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-02T02:49:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:49:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:49:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:49:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-02T02:49:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T02:49:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:49:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:49:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T02:49:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T02:49:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:49:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:49:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:49:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T02:49:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T02:49:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:49:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T02:49:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:49:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T02:49:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T02:49:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:49:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:49:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T02:49:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T02:49:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:49:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:49:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:49:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T02:49:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:49:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:49:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T02:49:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T02:49:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T02:49:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:49:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:49:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:49:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-02T02:49:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:49:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:49:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:49:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-02T02:49:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T02:49:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:49:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:49:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T02:49:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T02:49:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:49:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:49:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:49:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T02:49:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T02:49:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:49:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T02:49:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T02:49:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T02:49:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T02:49:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:49:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:49:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T02:49:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T02:49:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:49:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:49:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T02:49:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T02:49:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:49:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:49:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T02:49:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T02:49:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T02:49:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T02:49:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T02:49:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T02:49:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-02T02:49:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:49:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:49:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:49:47.101 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:49:47.101 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 8 2023-12-02T02:49:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T02:49:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T02:49:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T02:49:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T02:49:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:49:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:49:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T02:49:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T02:49:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:49:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:49:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:49:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T02:49:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T02:49:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:49:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T02:49:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:49:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T02:49:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T02:49:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:49:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:49:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T02:49:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T02:49:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:49:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:49:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:49:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T02:49:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:49:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:49:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T02:49:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T02:49:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T02:49:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:49:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:49:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:49:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T02:49:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:49:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:49:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:49:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-02T02:49:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T02:49:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:49:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:49:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T02:49:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T02:49:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:49:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:49:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:49:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T02:49:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T02:49:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:49:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T02:49:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:49:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T02:49:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T02:49:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:49:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:49:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T02:49:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T02:49:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:49:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:49:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:49:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T02:49:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:49:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:49:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T02:49:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T02:49:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T02:49:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:49:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:49:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:49:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-02T02:49:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:49:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:49:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:49:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-02T02:49:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T02:49:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:49:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:49:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T02:49:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T02:49:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:49:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:49:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:49:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T02:49:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T02:49:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:49:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T02:49:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:49:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T02:49:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T02:49:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:49:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:49:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T02:49:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T02:49:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:49:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:49:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:49:55.307 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T02:49:55.307 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:49:55.307 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:49:55.307 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T02:49:55.307 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T02:49:55.307 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T02:49:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:49:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:49:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:49:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-02T02:49:55.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:49:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:49:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:49:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:49:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-02T02:49:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T02:49:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:49:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:49:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T02:49:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T02:49:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:49:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:49:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:49:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T02:49:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T02:49:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:49:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T02:49:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T02:49:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T02:49:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T02:49:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:49:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:49:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T02:49:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T02:49:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:49:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:49:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:49:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T02:49:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T02:49:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:49:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:49:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T02:49:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T02:49:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T02:49:55.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T02:49:55.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T02:49:55.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T02:49:55.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-02T02:49:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:49:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:49:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:49:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:49:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 8 2023-12-02T02:49:58.881 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3848835403882962) > weight (0.25) 2023-12-02T02:49:58.881 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-02T02:49:58.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-02T02:49:58.882 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-02T02:49:58.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi057.stderr:2023-12-02T02:49:58.879+0000 7f9505f11640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 91639) UID: 1000 2023-12-02T02:50:01.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr:2023-12-02T02:50:01.413+0000 7fe3a9e0f640 -1 rbd::mirror::Mirror: 0x5630e662bd40 update_pool_replayers: restarting failed pool replayer for uuid: 06fbe0df-d787-44c2-aede-d6f6ace95f83 cluster: cluster2 client: client.mirror 2023-12-02T02:50:01.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr:2023-12-02T02:50:01.492+0000 7fe3a9e0f640 -1 rbd::mirror::Mirror: 0x5630e662bd40 update_pool_replayers: restarting failed pool replayer for uuid: 23f8ff71-737e-4eab-b981-22885d6c63ed cluster: cluster2 client: client.mirror 2023-12-02T02:50:01.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi057.stderr:2023-12-02T02:50:01.550+0000 7f2964e0f640 -1 rbd::mirror::Mirror: 0x561c9e30fd40 update_pool_replayers: restarting failed pool replayer for uuid: 06fbe0df-d787-44c2-aede-d6f6ace95f83 cluster: cluster2 client: client.mirror 2023-12-02T02:50:01.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi057.stderr:2023-12-02T02:50:01.552+0000 7f800102e640 -1 rbd::mirror::Mirror: 0x556840e9dd40 update_pool_replayers: restarting failed pool replayer for uuid: 06fbe0df-d787-44c2-aede-d6f6ace95f83 cluster: cluster2 client: client.mirror 2023-12-02T02:50:01.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi057.stderr:2023-12-02T02:50:01.607+0000 7f2964e0f640 -1 rbd::mirror::Mirror: 0x561c9e30fd40 update_pool_replayers: restarting failed pool replayer for uuid: 23f8ff71-737e-4eab-b981-22885d6c63ed cluster: cluster2 client: client.mirror 2023-12-02T02:50:01.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi057.stderr:2023-12-02T02:50:01.611+0000 7f800102e640 -1 rbd::mirror::Mirror: 0x556840e9dd40 update_pool_replayers: restarting failed pool replayer for uuid: 23f8ff71-737e-4eab-b981-22885d6c63ed cluster: cluster2 client: client.mirror 2023-12-02T02:50:02.883 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-02T02:50:02.883 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T02:50:02.883 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-02T02:50:02.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-02T02:50:02.883 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-02T02:50:02.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-02T02:50:02.886 INFO:tasks.rbd_mirror_thrash:waiting for 94 secs before thrashing 2023-12-02T02:50:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T02:50:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T02:50:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:50:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T02:50:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T02:50:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:50:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:50:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T02:50:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T02:50:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:50:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:50:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:50:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T02:50:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T02:50:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:50:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:50:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T02:50:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:50:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T02:50:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T02:50:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:50:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:50:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T02:50:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T02:50:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:50:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:50:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:50:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:50:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T02:50:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:50:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:50:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T02:50:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T02:50:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T02:50:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:50:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:50:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:50:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T02:50:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:50:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:50:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:50:03.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:50:03.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-02T02:50:03.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T02:50:03.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:50:03.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:50:03.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T02:50:03.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T02:50:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:50:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:50:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:50:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T02:50:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T02:50:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:50:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:50:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T02:50:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:50:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T02:50:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T02:50:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:50:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:50:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T02:50:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T02:50:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:50:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:50:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:50:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:50:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T02:50:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:50:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:50:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T02:50:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T02:50:03.609 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T02:50:03.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:50:03.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:50:03.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:50:03.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-02T02:50:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:50:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:50:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:50:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:50:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-02T02:50:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T02:50:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:50:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:50:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T02:50:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T02:50:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:50:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:50:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:50:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T02:50:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T02:50:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:50:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:50:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T02:50:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:50:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T02:50:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T02:50:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:50:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:50:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T02:50:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T02:50:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:50:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:50:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:50:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:50:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T02:50:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:50:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:50:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T02:50:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T02:50:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T02:50:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:50:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:50:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:50:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-02T02:50:03.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T02:50:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T02:50:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T02:50:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T02:50:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T02:50:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T02:50:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T02:50:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:50:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ clean_snap_name= 2023-12-02T02:50:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ seq 1 10 2023-12-02T02:50:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:50:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror test 2023-12-02T02:50:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:50:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:50:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:50:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T02:50:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=11 2023-12-02T02:50:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T02:50:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-02T02:50:04.040 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 93542400 bytes 2023-12-02T02:50:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 145157120 bytes 2023-12-02T02:50:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 162669568 bytes 2023-12-02T02:50:04.318 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 203888640 bytes 2023-12-02T02:50:04.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 290381824 bytes 2023-12-02T02:50:04.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 331431936 bytes 2023-12-02T02:50:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 369902592 bytes 2023-12-02T02:50:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 472199168 bytes 2023-12-02T02:50:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 491311104 bytes 2023-12-02T02:50:05.060 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 535840768 bytes 2023-12-02T02:50:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 554245120 bytes 2023-12-02T02:50:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 613922816 bytes 2023-12-02T02:50:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 640112640 bytes 2023-12-02T02:50:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 680014848 bytes 2023-12-02T02:50:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 773532672 bytes 2023-12-02T02:50:05.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 877655040 bytes 2023-12-02T02:50:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 936345600 bytes 2023-12-02T02:50:05.852 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 946782208 bytes 2023-12-02T02:50:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 1013357568 bytes 2023-12-02T02:50:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 1036772352 bytes 2023-12-02T02:50:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 1041664000 bytes 2023-12-02T02:50:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 1056177152 bytes 2023-12-02T02:50:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 1077382144 bytes 2023-12-02T02:50:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1077965824 bytes 2023-12-02T02:50:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1124107264 bytes 2023-12-02T02:50:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1199998976 bytes 2023-12-02T02:50:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1228556288 bytes 2023-12-02T02:50:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1293476864 bytes 2023-12-02T02:50:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1350533120 bytes 2023-12-02T02:50:06.882 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9481780479145749) > weight (0.25) 2023-12-02T02:50:06.882 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.38821555761376014) > weight (0.25) 2023-12-02T02:50:06.882 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-02T02:50:06.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-02T02:50:06.882 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-02T02:50:06.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi057.stderr:2023-12-02T02:50:06.880+0000 7efed3c58640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 91838) UID: 1000 2023-12-02T02:50:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1424023552 bytes 2023-12-02T02:50:07.085 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1486003200 bytes 2023-12-02T02:50:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1545947136 bytes 2023-12-02T02:50:07.380 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1668717568 bytes 2023-12-02T02:50:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1797842944 bytes 2023-12-02T02:50:07.639 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1821914112 bytes 2023-12-02T02:50:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1851572224 bytes 2023-12-02T02:50:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1861993472 bytes 2023-12-02T02:50:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1874109440 bytes 2023-12-02T02:50:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1887072256 bytes 2023-12-02T02:50:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1889856512 bytes 2023-12-02T02:50:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1905447936 bytes 2023-12-02T02:50:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1940921344 bytes 2023-12-02T02:50:08.178 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1990653952 bytes 2023-12-02T02:50:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1994149888 bytes 2023-12-02T02:50:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 2018698240 bytes 2023-12-02T02:50:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 2023311360 bytes 2023-12-02T02:50:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 2040200192 bytes 2023-12-02T02:50:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2069091328 bytes 2023-12-02T02:50:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2080166912 bytes 2023-12-02T02:50:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2100802560 bytes 2023-12-02T02:50:10.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2127639552 bytes 2023-12-02T02:50:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2170252288 bytes 2023-12-02T02:50:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2245754880 bytes 2023-12-02T02:50:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2277726208 bytes 2023-12-02T02:50:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2311907328 bytes 2023-12-02T02:50:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2329845760 bytes 2023-12-02T02:50:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2348545024 bytes 2023-12-02T02:50:11.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2370169856 bytes 2023-12-02T02:50:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2404939776 bytes 2023-12-02T02:50:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2423409664 bytes 2023-12-02T02:50:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2473339904 bytes 2023-12-02T02:50:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2507381760 bytes 2023-12-02T02:50:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2537647104 bytes 2023-12-02T02:50:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2562208768 bytes 2023-12-02T02:50:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2598635520 bytes 2023-12-02T02:50:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2636675072 bytes 2023-12-02T02:50:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2659024896 bytes 2023-12-02T02:50:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2725651456 bytes 2023-12-02T02:50:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 2799816704 bytes 2023-12-02T02:50:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 2825269248 bytes 2023-12-02T02:50:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 2872999936 bytes 2023-12-02T02:50:13.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 2929015808 bytes 2023-12-02T02:50:13.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 2944631808 bytes 2023-12-02T02:50:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 2969341952 bytes 2023-12-02T02:50:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3023214592 bytes 2023-12-02T02:50:13.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3117576192 bytes 2023-12-02T02:50:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3127608320 bytes 2023-12-02T02:50:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3150986240 bytes 2023-12-02T02:50:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3163143168 bytes 2023-12-02T02:50:14.190 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3168534528 bytes 2023-12-02T02:50:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3243523072 bytes 2023-12-02T02:50:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8200 IOs, 3340273664 bytes 2023-12-02T02:50:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8300 IOs, 3405377536 bytes 2023-12-02T02:50:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8400 IOs, 3469220864 bytes 2023-12-02T02:50:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8500 IOs, 3491548160 bytes 2023-12-02T02:50:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T02:50:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T02:50:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T02:50:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:50:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-02T02:50:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:50:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:50:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:50:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:50:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:50:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:50:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:50:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:50:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T02:50:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:50:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:50:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:50:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:50:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:50:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:50:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T02:50:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T02:50:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:50:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:50:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:50:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T02:50:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ tee /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:50:15.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:test: 2023-12-02T02:50:15.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: global_id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:50:15.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: down+unknown 2023-12-02T02:50:15.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: status not found 2023-12-02T02:50:15.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02T02:50:15.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: peer_sites: 2023-12-02T02:50:15.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: name: cluster2 2023-12-02T02:50:15.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: down+unknown 2023-12-02T02:50:15.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: status not found 2023-12-02T02:50:15.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02T02:50:15.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:50:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 1 2023-12-02T02:50:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:50:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 2 2023-12-02T02:50:15.883 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-02T02:50:15.883 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T02:50:15.884 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-02T02:50:15.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-02T02:50:15.884 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-02T02:50:15.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-02T02:50:15.887 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2023-12-02T02:50:17.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T02:50:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:50:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:50:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:50:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:50:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:50:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:50:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T02:50:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T02:50:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:50:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:50:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:50:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T02:50:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ tee /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:50:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:test: 2023-12-02T02:50:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: global_id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:50:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: down+unknown 2023-12-02T02:50:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: status not found 2023-12-02T02:50:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02T02:50:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: peer_sites: 2023-12-02T02:50:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: name: cluster2 2023-12-02T02:50:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: down+unknown 2023-12-02T02:50:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: status not found 2023-12-02T02:50:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02T02:50:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:50:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 1 2023-12-02T02:50:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:50:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 4 2023-12-02T02:50:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T02:50:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:50:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:50:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:50:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:50:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:50:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:50:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T02:50:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T02:50:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:50:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:50:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:50:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T02:50:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ tee /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:50:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:test: 2023-12-02T02:50:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: global_id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:50:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: down+unknown 2023-12-02T02:50:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: status not found 2023-12-02T02:50:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02T02:50:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: peer_sites: 2023-12-02T02:50:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: name: cluster2 2023-12-02T02:50:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: down+unknown 2023-12-02T02:50:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: status not found 2023-12-02T02:50:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02T02:50:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:50:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 1 2023-12-02T02:50:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:50:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 8 2023-12-02T02:50:29.991 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T02:50:29.991 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:50:29.991 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:50:29.991 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:50:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:50:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:50:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:50:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T02:50:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T02:50:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:50:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:50:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:50:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T02:50:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ tee /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:50:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:test: 2023-12-02T02:50:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: global_id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:50:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: down+unknown 2023-12-02T02:50:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: status not found 2023-12-02T02:50:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02T02:50:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: peer_sites: 2023-12-02T02:50:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: name: cluster2 2023-12-02T02:50:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: down+unknown 2023-12-02T02:50:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: status not found 2023-12-02T02:50:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02T02:50:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:50:30.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 1 2023-12-02T02:50:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:50:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 8 2023-12-02T02:50:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T02:50:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:50:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:50:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:50:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:50:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:50:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:50:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T02:50:38.050 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T02:50:38.050 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:50:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:50:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:50:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T02:50:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ tee /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:50:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:test: 2023-12-02T02:50:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: global_id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:50:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: up+replaying 2023-12-02T02:50:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701485377,"remote_snapshot_timestamp":1701485377,"replay_state":"idle"} 2023-12-02T02:50:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: service: mirror.2 on smithi057 2023-12-02T02:50:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02 02:50:31 2023-12-02T02:50:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: peer_sites: 2023-12-02T02:50:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: name: cluster2 2023-12-02T02:50:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: up+stopped 2023-12-02T02:50:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: local image is primary 2023-12-02T02:50:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02 02:50:31 2023-12-02T02:50:38.107 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:50:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T02:50:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ description: .*' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:50:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701485377,"remote_snapshot_timestamp":1701485377,"replay_state":"idle"} 2023-12-02T02:50:38.110 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T02:50:38.110 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo up+replaying 2023-12-02T02:50:38.110 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^up+' 2023-12-02T02:50:38.111 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:up+replaying 2023-12-02T02:50:38.111 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:50:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.2 on smithi057 2023-12-02T02:50:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:50:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local 'last_update=2023-12-02 02:50:31' 2023-12-02T02:50:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-02 02:50:31' 2023-12-02T02:50:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:50:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:50:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:50:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:50:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local 'prev_last_update=2023-12-02 02:50:31' 2023-12-02T02:50:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-02T02:50:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror.mirror_status 2023-12-02T02:50:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:50:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror.mirror_status 2023-12-02T02:50:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-02T02:50:38.153 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T02:50:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local last_update state 2023-12-02T02:50:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-02T02:50:38.220 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ last_update='2023-12-02 02:50:31' 2023-12-02T02:50:38.221 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-02T02:50:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ state=up+replaying 2023-12-02T02:50:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo up+replaying 2023-12-02T02:50:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep up+replaying 2023-12-02T02:50:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:up+replaying 2023-12-02T02:50:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:50:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:50:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap1 2023-12-02T02:50:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror test snap1 2023-12-02T02:50:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:50:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:50:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:50:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap1 2023-12-02T02:50:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2023-12-02T02:50:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-02T02:50:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:50:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:50:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:50:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-02T02:50:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:50:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:50:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:50:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T02:50:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T02:50:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:50:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:50:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-02T02:50:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:50:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:50:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:50:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T02:50:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T02:50:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T02:50:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-02T02:50:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T02:50:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T02:50:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:50:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T02:50:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T02:50:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T02:50:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:50:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T02:50:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T02:50:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T02:50:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:50:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T02:50:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T02:50:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:50:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:50:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T02:50:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T02:50:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:50:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:50:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:50:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T02:50:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T02:50:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:50:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:50:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T02:50:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:50:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T02:50:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T02:50:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:50:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:50:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T02:50:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T02:50:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:50:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:50:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:50:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:50:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T02:50:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:50:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:50:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T02:50:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T02:50:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T02:50:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:50:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:50:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:50:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T02:50:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:50:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:50:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:50:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:50:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-02T02:50:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T02:50:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:50:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:50:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T02:50:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T02:50:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:50:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:50:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:50:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T02:50:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T02:50:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:50:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:50:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T02:50:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:50:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T02:50:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T02:50:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:50:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:50:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T02:50:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T02:50:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:50:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:50:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:50:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:50:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T02:50:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:50:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:50:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T02:50:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T02:50:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T02:50:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:50:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:50:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:50:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-02T02:50:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:50:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:50:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:50:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:50:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-02T02:50:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T02:50:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:50:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:50:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T02:50:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T02:50:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:50:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:50:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:50:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T02:50:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T02:50:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:50:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:50:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T02:50:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:50:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T02:50:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T02:50:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:50:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:50:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T02:50:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T02:50:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:50:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:50:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:50:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:50:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T02:50:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:50:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:50:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T02:50:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T02:50:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T02:50:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:50:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:50:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:50:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-02T02:50:46.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T02:50:46.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T02:50:46.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T02:50:46.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T02:50:46.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T02:50:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T02:50:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T02:50:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:50:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-02T02:50:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T02:50:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:50:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:50:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:50:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T02:50:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T02:50:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-02T02:50:46.707 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T02:50:46.707 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:50:46.707 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:50:46.707 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:50:46.707 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-test.status 2023-12-02T02:50:46.707 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-test.status 2023-12-02T02:50:46.707 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:50:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:50:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.status 2023-12-02T02:50:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.status 2023-12-02T02:50:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:50:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:50:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-02T02:50:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:50:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:50:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:50:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-02T02:50:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 6 2023-12-02T02:50:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:50:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:50:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:50:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:50:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:50:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-02T02:50:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:50:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T02:50:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=6 2023-12-02T02:50:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:50:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:50:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T02:50:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:50:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:50:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:50:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:50:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:50:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:50:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:50:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:50:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=4 2023-12-02T02:50:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 6 = 4 2023-12-02T02:50:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:50:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T02:50:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:50:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:50:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:50:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:50:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:50:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:50:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:50:48.167 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:50:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=4 2023-12-02T02:50:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 6 = 4 2023-12-02T02:50:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:50:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T02:50:48.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:50:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:50:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:50:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:50:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:50:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:50:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:50:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:50:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:50:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:50:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:50:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:50:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:50:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 6 = '' 2023-12-02T02:50:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:50:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1.6 2023-12-02T02:50:50.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:50:50.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:50:50.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:50:50.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:50:50.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:50:50.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:50:50.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:50:50.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:50:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=6 2023-12-02T02:50:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 6 = 6 2023-12-02T02:50:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:50:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2023-12-02T02:50:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:50:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:50:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:50:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap1 2023-12-02T02:50:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T02:50:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T02:50:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:50:51.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2023-12-02T02:50:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'test': 2023-12-02T02:50:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 512 MiB in 128 objects 2023-12-02T02:50:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T02:50:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T02:50:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 10b949c95412 2023-12-02T02:50:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.10b949c95412 2023-12-02T02:50:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T02:50:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T02:50:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T02:50:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T02:50:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 02:50:03 2023 2023-12-02T02:50:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 02:50:03 2023 2023-12-02T02:50:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 02:50:03 2023 2023-12-02T02:50:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T02:50:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T02:50:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T02:50:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:50:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T02:50:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:50:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T02:50:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap1 2023-12-02T02:50:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:50:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:50:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap1 2023-12-02T02:50:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:50:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:50:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:50:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:50:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:50:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:50:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:50:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:50:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:50:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:50:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:50:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ clean_snap_name=snap1-clean 2023-12-02T02:50:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror test snap1-clean 2023-12-02T02:50:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:50:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:50:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:50:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap1-clean 2023-12-02T02:50:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2023-12-02T02:50:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:50:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror test 2023-12-02T02:50:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:50:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:50:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:50:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T02:50:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=8 2023-12-02T02:50:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T02:50:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-02T02:50:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 28139520 bytes 2023-12-02T02:50:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 44840960 bytes 2023-12-02T02:50:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 71050240 bytes 2023-12-02T02:50:54.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 123228160 bytes 2023-12-02T02:50:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 144988160 bytes 2023-12-02T02:50:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 181532672 bytes 2023-12-02T02:50:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 182141952 bytes 2023-12-02T02:50:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 215083008 bytes 2023-12-02T02:50:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 317640704 bytes 2023-12-02T02:50:55.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 324647936 bytes 2023-12-02T02:50:55.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 371719168 bytes 2023-12-02T02:50:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 419152896 bytes 2023-12-02T02:50:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 421927936 bytes 2023-12-02T02:50:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 496314368 bytes 2023-12-02T02:50:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 519970816 bytes 2023-12-02T02:50:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 552376320 bytes 2023-12-02T02:50:55.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 566987776 bytes 2023-12-02T02:50:56.066 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 591111168 bytes 2023-12-02T02:50:56.190 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 664809472 bytes 2023-12-02T02:50:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 744813568 bytes 2023-12-02T02:50:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 745410560 bytes 2023-12-02T02:50:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 764301312 bytes 2023-12-02T02:50:56.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 829395968 bytes 2023-12-02T02:50:56.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 867617792 bytes 2023-12-02T02:50:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 941856768 bytes 2023-12-02T02:50:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 972855296 bytes 2023-12-02T02:50:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1007983616 bytes 2023-12-02T02:50:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1036156928 bytes 2023-12-02T02:50:57.140 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1078925312 bytes 2023-12-02T02:50:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1131764736 bytes 2023-12-02T02:50:57.309 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1142575104 bytes 2023-12-02T02:50:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1196646400 bytes 2023-12-02T02:50:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1292081152 bytes 2023-12-02T02:50:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1330083840 bytes 2023-12-02T02:50:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1392553984 bytes 2023-12-02T02:50:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1437157376 bytes 2023-12-02T02:50:57.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1479464960 bytes 2023-12-02T02:50:58.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1505455104 bytes 2023-12-02T02:50:58.155 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1555357696 bytes 2023-12-02T02:50:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1621201920 bytes 2023-12-02T02:50:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1667838976 bytes 2023-12-02T02:50:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1707400192 bytes 2023-12-02T02:50:58.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1755396096 bytes 2023-12-02T02:50:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1788143616 bytes 2023-12-02T02:50:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1827899392 bytes 2023-12-02T02:50:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1877632000 bytes 2023-12-02T02:50:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1905838080 bytes 2023-12-02T02:50:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 1945966592 bytes 2023-12-02T02:50:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2001999872 bytes 2023-12-02T02:50:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2023531520 bytes 2023-12-02T02:50:59.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2079099904 bytes 2023-12-02T02:50:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2147472384 bytes 2023-12-02T02:50:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2148081664 bytes 2023-12-02T02:50:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2178544640 bytes 2023-12-02T02:50:59.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2227377152 bytes 2023-12-02T02:51:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2298764288 bytes 2023-12-02T02:51:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2385991680 bytes 2023-12-02T02:51:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2446329856 bytes 2023-12-02T02:51:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2469551104 bytes 2023-12-02T02:51:00.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2477417472 bytes 2023-12-02T02:51:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2485820416 bytes 2023-12-02T02:51:00.551 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2508131328 bytes 2023-12-02T02:51:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2532911104 bytes 2023-12-02T02:51:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2579395584 bytes 2023-12-02T02:51:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2638297088 bytes 2023-12-02T02:51:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2712177664 bytes 2023-12-02T02:51:01.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2763804672 bytes 2023-12-02T02:51:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2774516736 bytes 2023-12-02T02:51:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 2793080832 bytes 2023-12-02T02:51:01.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 2868527104 bytes 2023-12-02T02:51:01.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 2884128768 bytes 2023-12-02T02:51:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 2911414272 bytes 2023-12-02T02:51:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 2974691328 bytes 2023-12-02T02:51:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 2991897600 bytes 2023-12-02T02:51:01.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3039384576 bytes 2023-12-02T02:51:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3063806976 bytes 2023-12-02T02:51:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3122995200 bytes 2023-12-02T02:51:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3123564544 bytes 2023-12-02T02:51:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3149026304 bytes 2023-12-02T02:51:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3187710976 bytes 2023-12-02T02:51:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3204115456 bytes 2023-12-02T02:51:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8200 IOs, 3228395520 bytes 2023-12-02T02:51:02.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T02:51:02.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T02:51:02.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T02:51:02.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:51:02.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-02T02:51:02.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:51:02.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:51:02.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:51:02.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:51:02.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:51:02.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:51:02.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:51:02.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:51:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T02:51:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:51:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:51:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:51:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:51:03.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:51:03.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:51:03.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T02:51:03.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T02:51:03.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:51:03.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:51:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:51:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T02:51:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ tee /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:51:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:test: 2023-12-02T02:51:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: global_id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:51:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: up+replaying 2023-12-02T02:51:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: replaying, {"bytes_per_second":17873988.27,"bytes_per_snapshot":268109824.0,"last_snapshot_bytes":536219648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701485447,"remote_snapshot_timestamp":1701485447,"replay_state":"idle"} 2023-12-02T02:51:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: service: mirror.2 on smithi057 2023-12-02T02:51:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02 02:51:01 2023-12-02T02:51:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: peer_sites: 2023-12-02T02:51:03.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: name: cluster2 2023-12-02T02:51:03.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: up+stopped 2023-12-02T02:51:03.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: local image is primary 2023-12-02T02:51:03.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02 02:51:01 2023-12-02T02:51:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:51:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T02:51:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ description: .*' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:51:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":17873988.27,"bytes_per_snapshot":268109824.0,"last_snapshot_bytes":536219648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701485447,"remote_snapshot_timestamp":1701485447,"replay_state":"idle"} 2023-12-02T02:51:03.484 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T02:51:03.484 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo up+replaying 2023-12-02T02:51:03.484 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^up+' 2023-12-02T02:51:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:up+replaying 2023-12-02T02:51:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:51:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.2 on smithi057 2023-12-02T02:51:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:51:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local 'last_update=2023-12-02 02:51:01' 2023-12-02T02:51:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-02 02:51:01' 2023-12-02T02:51:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:51:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:51:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:51:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:51:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local 'prev_last_update=2023-12-02 02:51:01' 2023-12-02T02:51:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-02T02:51:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror.mirror_status 2023-12-02T02:51:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:51:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror.mirror_status 2023-12-02T02:51:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-02T02:51:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T02:51:03.550 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local last_update state 2023-12-02T02:51:03.550 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-02T02:51:03.551 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ last_update='2023-12-02 02:51:01' 2023-12-02T02:51:03.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-02T02:51:03.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ state=up+replaying 2023-12-02T02:51:03.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo up+replaying 2023-12-02T02:51:03.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep up+replaying 2023-12-02T02:51:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:up+replaying 2023-12-02T02:51:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:51:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:51:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap2 2023-12-02T02:51:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror test snap2 2023-12-02T02:51:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:51:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:51:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:51:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap2 2023-12-02T02:51:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2023-12-02T02:51:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-02T02:51:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:51:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:51:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:51:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-02T02:51:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:51:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:51:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:51:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T02:51:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T02:51:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:51:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:51:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-02T02:51:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:51:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:51:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:51:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T02:51:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T02:51:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T02:51:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-02T02:51:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T02:51:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:51:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T02:51:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T02:51:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T02:51:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T02:51:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:51:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T02:51:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T02:51:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T02:51:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:51:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T02:51:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T02:51:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:51:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:51:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T02:51:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T02:51:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:51:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:51:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:51:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T02:51:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T02:51:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:51:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:51:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T02:51:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:51:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T02:51:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T02:51:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:51:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:51:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T02:51:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T02:51:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:51:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:51:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:51:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:51:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T02:51:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:51:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:51:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T02:51:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T02:51:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T02:51:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:51:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:51:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:51:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T02:51:32.535 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:51:32.535 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:51:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:51:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:51:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-02T02:51:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T02:51:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:51:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:51:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T02:51:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T02:51:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:51:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:51:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:51:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T02:51:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T02:51:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:51:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:51:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T02:51:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:51:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T02:51:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T02:51:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:51:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:51:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T02:51:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T02:51:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:51:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:51:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:51:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:51:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T02:51:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:51:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:51:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T02:51:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T02:51:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T02:51:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:51:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:51:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:51:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-02T02:51:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:51:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:51:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:51:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:51:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-02T02:51:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T02:51:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:51:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:51:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T02:51:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T02:51:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:51:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:51:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:51:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T02:51:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T02:51:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:51:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:51:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T02:51:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:51:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T02:51:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T02:51:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:51:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:51:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T02:51:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T02:51:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:51:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:51:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:51:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:51:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T02:51:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:51:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:51:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T02:51:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T02:51:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T02:51:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:51:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:51:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:51:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-02T02:51:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T02:51:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T02:51:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T02:51:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T02:51:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T02:51:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T02:51:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T02:51:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:51:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-02T02:51:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T02:51:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:51:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:51:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:51:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T02:51:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T02:51:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-02T02:51:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T02:51:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:51:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:51:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:51:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-test.status 2023-12-02T02:51:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-test.status 2023-12-02T02:51:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:51:32.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:51:32.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.status 2023-12-02T02:51:32.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.status 2023-12-02T02:51:32.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:51:32.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:51:32.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-02T02:51:32.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:51:32.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:51:32.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:51:32.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-02T02:51:34.451 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 9 2023-12-02T02:51:34.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:51:34.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:51:34.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:51:34.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:51:34.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:51:34.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-02T02:51:34.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:51:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T02:51:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=9 2023-12-02T02:51:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:51:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:51:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T02:51:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:51:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:51:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:51:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:51:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:51:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:51:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:51:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:51:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:51:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:51:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:51:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:51:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:51:34.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 9 = '' 2023-12-02T02:51:34.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:51:34.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T02:51:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:51:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:51:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:51:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:51:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:51:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:51:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:51:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:51:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:51:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:51:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:51:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:51:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:51:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 9 = '' 2023-12-02T02:51:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:51:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T02:51:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:51:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:51:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:51:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:51:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:51:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:51:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:51:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:51:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:51:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:51:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:51:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:51:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:51:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 9 = '' 2023-12-02T02:51:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:51:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1.6 2023-12-02T02:51:36.887 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-02T02:51:36.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-02T02:51:36.888 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-12-02T02:51:36.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi057.stderr:2023-12-02T02:51:36.886+0000 7f7ffd6a7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 91622) UID: 1000 2023-12-02T02:51:37.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:51:37.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:51:37.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:51:37.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:51:37.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:51:37.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:51:37.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:51:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:51:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=9 2023-12-02T02:51:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 9 = 9 2023-12-02T02:51:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:51:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2023-12-02T02:51:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:51:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:51:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:51:37.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap2 2023-12-02T02:51:37.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T02:51:37.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T02:51:37.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:51:38.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2023-12-02T02:51:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'test': 2023-12-02T02:51:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 512 MiB in 128 objects 2023-12-02T02:51:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T02:51:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 4 2023-12-02T02:51:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 10b949c95412 2023-12-02T02:51:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.10b949c95412 2023-12-02T02:51:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T02:51:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T02:51:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T02:51:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T02:51:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 02:50:03 2023 2023-12-02T02:51:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 02:50:03 2023 2023-12-02T02:51:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 02:50:03 2023 2023-12-02T02:51:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T02:51:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T02:51:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T02:51:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:51:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T02:51:38.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:51:38.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snap1-clean ']' 2023-12-02T02:51:38.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap1-clean 2023-12-02T02:51:38.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:51:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:51:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap1-clean 2023-12-02T02:51:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:51:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:51:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:51:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:51:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:51:38.888 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-02T02:51:38.888 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T02:51:38.889 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-02T02:51:38.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-02T02:51:38.889 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-02T02:51:38.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-02T02:51:38.892 INFO:tasks.rbd_mirror_thrash:waiting for 93 secs before thrashing 2023-12-02T02:51:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:51:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:51:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:51:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:51:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:51:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:51:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap2 2023-12-02T02:51:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:51:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:51:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap2 2023-12-02T02:51:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:51:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:51:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:51:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:51:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:51:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:51:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:51:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:51:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:51:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:51:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:51:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ clean_snap_name=snap2-clean 2023-12-02T02:51:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror test snap2-clean 2023-12-02T02:51:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:51:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:51:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:51:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap2-clean 2023-12-02T02:51:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2023-12-02T02:51:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:51:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror test 2023-12-02T02:51:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:51:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:51:42.364 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:51:42.364 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T02:51:42.365 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=9 2023-12-02T02:51:42.365 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T02:51:42.365 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-02T02:51:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 12665856 bytes 2023-12-02T02:51:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 31515648 bytes 2023-12-02T02:51:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 83648512 bytes 2023-12-02T02:51:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 116179968 bytes 2023-12-02T02:51:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 135618560 bytes 2023-12-02T02:51:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 163158016 bytes 2023-12-02T02:51:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 231822336 bytes 2023-12-02T02:51:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 237149184 bytes 2023-12-02T02:51:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 237755392 bytes 2023-12-02T02:51:43.320 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 312722432 bytes 2023-12-02T02:51:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 357701632 bytes 2023-12-02T02:51:43.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 393670656 bytes 2023-12-02T02:51:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 493935616 bytes 2023-12-02T02:51:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 538833920 bytes 2023-12-02T02:51:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 560060416 bytes 2023-12-02T02:51:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 597471232 bytes 2023-12-02T02:51:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 650355712 bytes 2023-12-02T02:51:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 697217024 bytes 2023-12-02T02:51:44.276 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 767816704 bytes 2023-12-02T02:51:44.380 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 813975552 bytes 2023-12-02T02:51:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 842342400 bytes 2023-12-02T02:51:44.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 892479488 bytes 2023-12-02T02:51:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 906107904 bytes 2023-12-02T02:51:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 914383872 bytes 2023-12-02T02:51:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 961074176 bytes 2023-12-02T02:51:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 981000192 bytes 2023-12-02T02:51:44.993 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1057334272 bytes 2023-12-02T02:51:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1092286464 bytes 2023-12-02T02:51:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1155575808 bytes 2023-12-02T02:51:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1217777664 bytes 2023-12-02T02:51:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1226978304 bytes 2023-12-02T02:51:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1251367936 bytes 2023-12-02T02:51:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1317466112 bytes 2023-12-02T02:51:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1368059904 bytes 2023-12-02T02:51:45.755 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1373839360 bytes 2023-12-02T02:51:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1445276672 bytes 2023-12-02T02:51:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1459165184 bytes 2023-12-02T02:51:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1487132672 bytes 2023-12-02T02:51:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1507316736 bytes 2023-12-02T02:51:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1590365184 bytes 2023-12-02T02:51:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1644158976 bytes 2023-12-02T02:51:46.522 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1677365248 bytes 2023-12-02T02:51:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1764619264 bytes 2023-12-02T02:51:46.778 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1813096448 bytes 2023-12-02T02:51:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1876122624 bytes 2023-12-02T02:51:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1928368128 bytes 2023-12-02T02:51:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1935213568 bytes 2023-12-02T02:51:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 1935814656 bytes 2023-12-02T02:51:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2037606400 bytes 2023-12-02T02:51:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2128171008 bytes 2023-12-02T02:51:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2181891072 bytes 2023-12-02T02:51:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2251246592 bytes 2023-12-02T02:51:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2295702528 bytes 2023-12-02T02:51:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2329332736 bytes 2023-12-02T02:51:47.985 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2375599104 bytes 2023-12-02T02:51:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2476451840 bytes 2023-12-02T02:51:48.232 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2505082880 bytes 2023-12-02T02:51:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2571066368 bytes 2023-12-02T02:51:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2596058112 bytes 2023-12-02T02:51:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2678037504 bytes 2023-12-02T02:51:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2678648832 bytes 2023-12-02T02:51:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2697064448 bytes 2023-12-02T02:51:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2697674752 bytes 2023-12-02T02:51:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2713722880 bytes 2023-12-02T02:51:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2748948480 bytes 2023-12-02T02:51:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2838464512 bytes 2023-12-02T02:51:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2853490688 bytes 2023-12-02T02:51:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2899289088 bytes 2023-12-02T02:51:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 2899891200 bytes 2023-12-02T02:51:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 2946700288 bytes 2023-12-02T02:51:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 3036639232 bytes 2023-12-02T02:51:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 3049944064 bytes 2023-12-02T02:51:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 3137482752 bytes 2023-12-02T02:51:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3168188416 bytes 2023-12-02T02:51:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3201380352 bytes 2023-12-02T02:51:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3208435712 bytes 2023-12-02T02:51:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3230057472 bytes 2023-12-02T02:51:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3257239552 bytes 2023-12-02T02:51:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3305081856 bytes 2023-12-02T02:51:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3387539456 bytes 2023-12-02T02:51:50.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3444670464 bytes 2023-12-02T02:51:50.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8200 IOs, 3462676480 bytes 2023-12-02T02:51:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8300 IOs, 3498188800 bytes 2023-12-02T02:51:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8400 IOs, 3551449088 bytes 2023-12-02T02:51:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8500 IOs, 3577663488 bytes 2023-12-02T02:51:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8600 IOs, 3660845056 bytes 2023-12-02T02:51:51.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8700 IOs, 3686415360 bytes 2023-12-02T02:51:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8800 IOs, 3736777728 bytes 2023-12-02T02:51:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8900 IOs, 3782603776 bytes 2023-12-02T02:51:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T02:51:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T02:51:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T02:51:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:51:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-02T02:51:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:51:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:51:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:51:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:51:51.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:51:51.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:51:51.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:51:51.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:51:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T02:51:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:51:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:51:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:51:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:51:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:51:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:51:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T02:51:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T02:51:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:51:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:51:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:51:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T02:51:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ tee /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:51:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:test: 2023-12-02T02:51:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: global_id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:51:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: up+replaying 2023-12-02T02:51:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":268109824.0,"last_snapshot_bytes":536219648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701485447,"remote_snapshot_timestamp":1701485447,"replay_state":"idle"} 2023-12-02T02:51:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: service: mirror.2 on smithi057 2023-12-02T02:51:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02 02:51:31 2023-12-02T02:51:52.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: peer_sites: 2023-12-02T02:51:52.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: name: cluster2 2023-12-02T02:51:52.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: up+stopped 2023-12-02T02:51:52.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: local image is primary 2023-12-02T02:51:52.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02 02:51:31 2023-12-02T02:51:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:51:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T02:51:52.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ description: .*' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:51:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":268109824.0,"last_snapshot_bytes":536219648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701485447,"remote_snapshot_timestamp":1701485447,"replay_state":"idle"} 2023-12-02T02:51:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T02:51:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo up+replaying 2023-12-02T02:51:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^up+' 2023-12-02T02:51:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:up+replaying 2023-12-02T02:51:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:51:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.2 on smithi057 2023-12-02T02:51:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:51:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local 'last_update=2023-12-02 02:51:31' 2023-12-02T02:51:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-02 02:51:31' 2023-12-02T02:51:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:51:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:51:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:51:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:51:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local 'prev_last_update=2023-12-02 02:51:31' 2023-12-02T02:51:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-02T02:51:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror.mirror_status 2023-12-02T02:51:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:51:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror.mirror_status 2023-12-02T02:51:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-02T02:51:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T02:51:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local last_update state 2023-12-02T02:51:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-02T02:51:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ last_update='2023-12-02 02:51:31' 2023-12-02T02:51:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-02T02:51:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ state=up+replaying 2023-12-02T02:51:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo up+replaying 2023-12-02T02:51:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep up+replaying 2023-12-02T02:51:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:up+replaying 2023-12-02T02:51:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:51:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:51:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap3 2023-12-02T02:51:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror test snap3 2023-12-02T02:51:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:51:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:51:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:51:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap3 2023-12-02T02:51:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2023-12-02T02:52:12.888 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.295527313359346) > weight (0.25) 2023-12-02T02:52:12.888 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7757368533057349) > weight (0.25) 2023-12-02T02:52:12.888 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8766098769974838) > weight (0.25) 2023-12-02T02:52:12.888 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5396944275511562) > weight (0.25) 2023-12-02T02:52:12.888 INFO:tasks.rbd_mirror_thrash:waiting for 70 secs before thrashing 2023-12-02T02:52:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-02T02:52:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:52:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:52:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:52:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-02T02:52:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:52:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:52:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:52:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T02:52:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T02:52:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:52:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:52:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-02T02:52:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:52:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:52:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:52:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T02:52:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T02:52:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T02:52:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-02T02:52:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T02:52:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T02:52:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:52:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T02:52:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T02:52:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T02:52:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:52:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T02:52:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T02:52:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T02:52:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:52:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T02:52:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T02:52:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:52:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:52:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T02:52:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T02:52:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:52:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:52:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:52:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T02:52:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T02:52:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:52:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:52:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T02:52:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:52:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T02:52:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T02:52:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:52:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:52:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T02:52:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T02:52:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:52:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:52:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:52:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:52:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T02:52:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:52:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:52:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T02:52:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T02:52:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T02:52:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:52:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:52:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:52:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T02:52:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:52:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:52:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:52:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:52:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-02T02:52:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T02:52:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:52:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:52:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T02:52:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T02:52:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:52:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:52:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:52:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T02:52:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T02:52:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:52:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:52:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T02:52:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:52:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T02:52:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T02:52:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:52:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:52:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T02:52:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T02:52:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:52:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:52:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:52:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:52:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T02:52:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:52:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:52:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T02:52:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T02:52:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T02:52:22.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:52:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:52:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:52:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-02T02:52:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:52:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:52:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:52:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:52:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-02T02:52:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T02:52:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:52:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:52:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T02:52:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T02:52:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:52:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:52:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:52:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T02:52:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T02:52:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:52:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:52:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T02:52:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:52:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T02:52:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T02:52:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:52:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:52:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T02:52:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T02:52:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:52:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:52:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:52:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:52:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T02:52:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:52:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:52:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T02:52:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T02:52:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T02:52:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:52:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:52:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:52:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-02T02:52:22.718 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T02:52:22.718 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T02:52:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T02:52:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T02:52:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T02:52:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T02:52:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T02:52:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:52:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-02T02:52:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T02:52:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:52:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:52:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:52:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T02:52:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T02:52:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-02T02:52:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T02:52:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:52:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:52:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:52:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-test.status 2023-12-02T02:52:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-test.status 2023-12-02T02:52:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:52:22.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:52:22.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.status 2023-12-02T02:52:22.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.status 2023-12-02T02:52:22.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:52:22.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:52:22.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-02T02:52:22.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:52:22.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:52:22.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:52:22.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-02T02:52:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 13 2023-12-02T02:52:24.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:52:24.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:52:24.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:52:24.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:52:24.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:52:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:52:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-02T02:52:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T02:52:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=13 2023-12-02T02:52:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:52:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:52:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T02:52:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:52:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:52:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:52:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:52:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:52:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:52:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:52:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:52:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:52:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:52:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:52:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:52:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:52:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 13 = '' 2023-12-02T02:52:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:52:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T02:52:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:52:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:52:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:52:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:52:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:52:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:52:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:52:25.228 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:52:25.228 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:52:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:52:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:52:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:52:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:52:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 13 = '' 2023-12-02T02:52:25.231 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:52:25.231 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T02:52:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:52:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:52:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:52:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:52:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:52:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:52:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:52:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:52:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:52:26.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:52:26.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:52:26.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:52:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:52:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 13 = '' 2023-12-02T02:52:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:52:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1.6 2023-12-02T02:52:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:52:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:52:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:52:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:52:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:52:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:52:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:52:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:52:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=13 2023-12-02T02:52:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 13 = 13 2023-12-02T02:52:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:52:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2023-12-02T02:52:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:52:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:52:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:52:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap3 2023-12-02T02:52:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T02:52:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T02:52:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:52:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2023-12-02T02:52:28.788 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'test': 2023-12-02T02:52:28.788 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 512 MiB in 128 objects 2023-12-02T02:52:28.788 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T02:52:28.788 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 6 2023-12-02T02:52:28.788 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 10b949c95412 2023-12-02T02:52:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.10b949c95412 2023-12-02T02:52:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T02:52:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T02:52:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T02:52:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T02:52:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 02:50:03 2023 2023-12-02T02:52:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 02:51:39 2023 2023-12-02T02:52:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 02:50:03 2023 2023-12-02T02:52:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T02:52:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T02:52:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T02:52:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:52:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T02:52:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:52:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snap2-clean ']' 2023-12-02T02:52:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap2-clean 2023-12-02T02:52:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:52:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:52:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap2-clean 2023-12-02T02:52:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:52:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:52:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:52:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:52:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:52:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:52:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:52:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:52:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:52:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:52:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:52:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap3 2023-12-02T02:52:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:52:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:52:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap3 2023-12-02T02:52:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:52:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:52:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:52:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:52:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:52:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:52:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:52:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:52:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:52:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:52:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:52:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ clean_snap_name=snap3-clean 2023-12-02T02:52:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror test snap3-clean 2023-12-02T02:52:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:52:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:52:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:52:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap3-clean 2023-12-02T02:52:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2023-12-02T02:52:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:52:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror test 2023-12-02T02:52:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:52:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:52:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:52:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T02:52:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=13 2023-12-02T02:52:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T02:52:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-02T02:52:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 23291904 bytes 2023-12-02T02:52:32.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 107353088 bytes 2023-12-02T02:52:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 161679360 bytes 2023-12-02T02:52:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 209871872 bytes 2023-12-02T02:52:33.104 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 265143296 bytes 2023-12-02T02:52:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 304906240 bytes 2023-12-02T02:52:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 351438848 bytes 2023-12-02T02:52:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 402785280 bytes 2023-12-02T02:52:33.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 485656576 bytes 2023-12-02T02:52:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 518341632 bytes 2023-12-02T02:52:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 585128960 bytes 2023-12-02T02:52:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 609429504 bytes 2023-12-02T02:52:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 683286528 bytes 2023-12-02T02:52:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 746333184 bytes 2023-12-02T02:52:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 789773312 bytes 2023-12-02T02:52:34.345 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 872095744 bytes 2023-12-02T02:52:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 958848000 bytes 2023-12-02T02:52:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 1008231424 bytes 2023-12-02T02:52:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 1035794432 bytes 2023-12-02T02:52:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 1123665920 bytes 2023-12-02T02:52:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 1155829760 bytes 2023-12-02T02:52:35.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 1190642688 bytes 2023-12-02T02:52:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 1225975808 bytes 2023-12-02T02:52:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1262697472 bytes 2023-12-02T02:52:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1263308800 bytes 2023-12-02T02:52:35.338 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1284924416 bytes 2023-12-02T02:52:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1308690432 bytes 2023-12-02T02:52:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1346992128 bytes 2023-12-02T02:52:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1420781568 bytes 2023-12-02T02:52:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1422359552 bytes 2023-12-02T02:52:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1492897792 bytes 2023-12-02T02:52:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1551264768 bytes 2023-12-02T02:52:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1567176704 bytes 2023-12-02T02:52:36.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1601636352 bytes 2023-12-02T02:52:36.154 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1624677376 bytes 2023-12-02T02:52:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1663620096 bytes 2023-12-02T02:52:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1685961728 bytes 2023-12-02T02:52:36.367 INFO:tasks.ceph.cluster2.osd.1.smithi057.stderr:2023-12-02T02:52:36.364+0000 7fbd9d384640 -1 reset not still connected to 0x55bdb4994d00 2023-12-02T02:52:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1717431296 bytes 2023-12-02T02:52:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1754048512 bytes 2023-12-02T02:52:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1801437184 bytes 2023-12-02T02:52:36.741 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1839521792 bytes 2023-12-02T02:52:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1877885952 bytes 2023-12-02T02:52:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1902919680 bytes 2023-12-02T02:52:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1979042816 bytes 2023-12-02T02:52:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 2020598784 bytes 2023-12-02T02:52:37.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 2070682624 bytes 2023-12-02T02:52:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 2103574528 bytes 2023-12-02T02:52:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2147859456 bytes 2023-12-02T02:52:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2190313472 bytes 2023-12-02T02:52:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2246770688 bytes 2023-12-02T02:52:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2266932224 bytes 2023-12-02T02:52:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2276898816 bytes 2023-12-02T02:52:37.890 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2310387712 bytes 2023-12-02T02:52:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2361883648 bytes 2023-12-02T02:52:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2383960064 bytes 2023-12-02T02:52:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2515294208 bytes 2023-12-02T02:52:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2518745088 bytes 2023-12-02T02:52:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2605275136 bytes 2023-12-02T02:52:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2663912448 bytes 2023-12-02T02:52:38.732 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2700329984 bytes 2023-12-02T02:52:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2708644864 bytes 2023-12-02T02:52:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2716849152 bytes 2023-12-02T02:52:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2737457152 bytes 2023-12-02T02:52:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2790419456 bytes 2023-12-02T02:52:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2813504512 bytes 2023-12-02T02:52:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2836209664 bytes 2023-12-02T02:52:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2858083328 bytes 2023-12-02T02:52:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2916622336 bytes 2023-12-02T02:52:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 2935616512 bytes 2023-12-02T02:52:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 2943963136 bytes 2023-12-02T02:52:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 3046591488 bytes 2023-12-02T02:52:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 3061458944 bytes 2023-12-02T02:52:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 3117116416 bytes 2023-12-02T02:52:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3156237312 bytes 2023-12-02T02:52:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3214084096 bytes 2023-12-02T02:52:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3281760256 bytes 2023-12-02T02:52:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3325789184 bytes 2023-12-02T02:52:40.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3379440640 bytes 2023-12-02T02:52:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3412330496 bytes 2023-12-02T02:52:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3477534720 bytes 2023-12-02T02:52:40.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3513028608 bytes 2023-12-02T02:52:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8200 IOs, 3552986112 bytes 2023-12-02T02:52:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8300 IOs, 3603236864 bytes 2023-12-02T02:52:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8400 IOs, 3620675584 bytes 2023-12-02T02:52:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8500 IOs, 3678339072 bytes 2023-12-02T02:52:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8600 IOs, 3697462272 bytes 2023-12-02T02:52:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8700 IOs, 3764350976 bytes 2023-12-02T02:52:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8800 IOs, 3770620928 bytes 2023-12-02T02:52:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8900 IOs, 3879471104 bytes 2023-12-02T02:52:41.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9000 IOs, 3980033024 bytes 2023-12-02T02:52:41.911 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9100 IOs, 3998308352 bytes 2023-12-02T02:52:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9200 IOs, 4013519872 bytes 2023-12-02T02:52:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9300 IOs, 4063670272 bytes 2023-12-02T02:52:42.168 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9400 IOs, 4076083200 bytes 2023-12-02T02:52:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9500 IOs, 4079613952 bytes 2023-12-02T02:52:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9600 IOs, 4096757760 bytes 2023-12-02T02:52:42.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9700 IOs, 4133045248 bytes 2023-12-02T02:52:42.530 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9800 IOs, 4212577280 bytes 2023-12-02T02:52:42.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9900 IOs, 4268705792 bytes 2023-12-02T02:52:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10000 IOs, 4271036416 bytes 2023-12-02T02:52:42.782 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10100 IOs, 4275275776 bytes 2023-12-02T02:52:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10200 IOs, 4280383488 bytes 2023-12-02T02:52:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10300 IOs, 4312507392 bytes 2023-12-02T02:52:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10400 IOs, 4402121728 bytes 2023-12-02T02:52:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10500 IOs, 4425635840 bytes 2023-12-02T02:52:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10600 IOs, 4448088064 bytes 2023-12-02T02:52:43.332 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10700 IOs, 4465617920 bytes 2023-12-02T02:52:43.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10800 IOs, 4536114176 bytes 2023-12-02T02:52:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10900 IOs, 4581687296 bytes 2023-12-02T02:52:43.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11000 IOs, 4641153024 bytes 2023-12-02T02:52:43.779 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11100 IOs, 4650138624 bytes 2023-12-02T02:52:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11200 IOs, 4683515904 bytes 2023-12-02T02:52:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11300 IOs, 4699274240 bytes 2023-12-02T02:52:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11400 IOs, 4722524160 bytes 2023-12-02T02:52:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11500 IOs, 4782969856 bytes 2023-12-02T02:52:44.244 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11600 IOs, 4807666688 bytes 2023-12-02T02:52:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11700 IOs, 4861342720 bytes 2023-12-02T02:52:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11800 IOs, 4902133760 bytes 2023-12-02T02:52:44.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11900 IOs, 4932717568 bytes 2023-12-02T02:52:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:12000 IOs, 4963496960 bytes 2023-12-02T02:52:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:12100 IOs, 4973796352 bytes 2023-12-02T02:52:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:12200 IOs, 5039234048 bytes 2023-12-02T02:52:44.963 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:12300 IOs, 5106176000 bytes 2023-12-02T02:52:45.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:12400 IOs, 5106791424 bytes 2023-12-02T02:52:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:12500 IOs, 5155684352 bytes 2023-12-02T02:52:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:12600 IOs, 5193341952 bytes 2023-12-02T02:52:45.320 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:12700 IOs, 5226016768 bytes 2023-12-02T02:52:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T02:52:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T02:52:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T02:52:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:52:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-02T02:52:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:52:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:52:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:52:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:52:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:52:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:52:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:52:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:52:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T02:52:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:52:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:52:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:52:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:52:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:52:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:52:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T02:52:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T02:52:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:52:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:52:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:52:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T02:52:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ tee /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:52:46.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:test: 2023-12-02T02:52:46.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: global_id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:52:46.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: up+replaying 2023-12-02T02:52:46.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: replaying, {"bytes_per_second":14959684.27,"bytes_per_snapshot":536584192.0,"last_snapshot_bytes":536745984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701485543,"remote_snapshot_timestamp":1701485543,"replay_state":"idle"} 2023-12-02T02:52:46.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: service: mirror.2 on smithi057 2023-12-02T02:52:46.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02 02:52:31 2023-12-02T02:52:46.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: peer_sites: 2023-12-02T02:52:46.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: name: cluster2 2023-12-02T02:52:46.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: up+stopped 2023-12-02T02:52:46.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: local image is primary 2023-12-02T02:52:46.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02 02:52:31 2023-12-02T02:52:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:52:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T02:52:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ description: .*' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:52:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":14959684.27,"bytes_per_snapshot":536584192.0,"last_snapshot_bytes":536745984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701485543,"remote_snapshot_timestamp":1701485543,"replay_state":"idle"} 2023-12-02T02:52:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T02:52:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo up+replaying 2023-12-02T02:52:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^up+' 2023-12-02T02:52:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:up+replaying 2023-12-02T02:52:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:52:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.2 on smithi057 2023-12-02T02:52:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:52:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local 'last_update=2023-12-02 02:52:31' 2023-12-02T02:52:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-02 02:52:31' 2023-12-02T02:52:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:52:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:52:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:52:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:52:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local 'prev_last_update=2023-12-02 02:52:31' 2023-12-02T02:52:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-02T02:52:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror.mirror_status 2023-12-02T02:52:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:52:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror.mirror_status 2023-12-02T02:52:46.571 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-02T02:52:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T02:52:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local last_update state 2023-12-02T02:52:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-02T02:52:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ last_update='2023-12-02 02:52:31' 2023-12-02T02:52:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-02T02:52:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ state=up+replaying 2023-12-02T02:52:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo up+replaying 2023-12-02T02:52:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep up+replaying 2023-12-02T02:52:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:up+replaying 2023-12-02T02:52:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:52:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:52:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap4 2023-12-02T02:52:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror test snap4 2023-12-02T02:52:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:52:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:52:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:52:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap4 2023-12-02T02:52:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2023-12-02T02:53:11.892 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-02T02:53:11.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-02T02:53:11.893 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-02T02:53:11.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi057.stderr:2023-12-02T02:53:11.891+0000 7f62e31b0640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 94458) UID: 1000 2023-12-02T02:53:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-02T02:53:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:53:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:53:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:53:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-02T02:53:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:53:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:53:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:53:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T02:53:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T02:53:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:53:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:53:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-02T02:53:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:53:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:53:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:53:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T02:53:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T02:53:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T02:53:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-02T02:53:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T02:53:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T02:53:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:53:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T02:53:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T02:53:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T02:53:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:53:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T02:53:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T02:53:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T02:53:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:53:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T02:53:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T02:53:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:53:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:53:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T02:53:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T02:53:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:53:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:53:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:53:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T02:53:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T02:53:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:53:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:53:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T02:53:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:53:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T02:53:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T02:53:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:53:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:53:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T02:53:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T02:53:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:53:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:53:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:53:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:53:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T02:53:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:53:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:53:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T02:53:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T02:53:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T02:53:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:53:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:53:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:53:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T02:53:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-02T02:53:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:53:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-02T02:53:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T02:53:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:53:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:53:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T02:53:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T02:53:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:53:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:53:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:53:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T02:53:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T02:53:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:53:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:53:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T02:53:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:53:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T02:53:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T02:53:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:53:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:53:15.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T02:53:15.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T02:53:15.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:53:15.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:53:15.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:53:15.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:53:15.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T02:53:15.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:53:15.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:53:15.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T02:53:15.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T02:53:15.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T02:53:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:53:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:53:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:53:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-02T02:53:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:53:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:53:15.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:53:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:53:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-02T02:53:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T02:53:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:53:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:53:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T02:53:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T02:53:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:53:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:53:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:53:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T02:53:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T02:53:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:53:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:53:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T02:53:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:53:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T02:53:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T02:53:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:53:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:53:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T02:53:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T02:53:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:53:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:53:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:53:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:53:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T02:53:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:53:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:53:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T02:53:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T02:53:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T02:53:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:53:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:53:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:53:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-02T02:53:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T02:53:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T02:53:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T02:53:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T02:53:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T02:53:15.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T02:53:15.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T02:53:15.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:53:15.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-02T02:53:15.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T02:53:15.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:53:15.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:53:15.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:53:15.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T02:53:15.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T02:53:15.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-02T02:53:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T02:53:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:53:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:53:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:53:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-test.status 2023-12-02T02:53:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-test.status 2023-12-02T02:53:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:53:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:53:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.status 2023-12-02T02:53:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.status 2023-12-02T02:53:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:53:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:53:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-02T02:53:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:53:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:53:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:53:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-02T02:53:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 17 2023-12-02T02:53:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:53:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:53:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:53:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:53:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:53:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-02T02:53:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:53:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T02:53:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=17 2023-12-02T02:53:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:53:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:53:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T02:53:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:53:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:53:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:53:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:53:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:53:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:53:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:53:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:53:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:53:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:53:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:53:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:53:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:53:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 17 = '' 2023-12-02T02:53:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:53:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T02:53:17.893 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-02T02:53:17.893 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T02:53:17.894 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-02T02:53:17.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-02T02:53:17.894 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-02T02:53:17.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-02T02:53:17.896 INFO:tasks.rbd_mirror_thrash:waiting for 112 secs before thrashing 2023-12-02T02:53:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:53:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:53:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:53:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:53:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:53:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:53:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:53:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:53:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:53:18.207 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:53:18.207 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:53:18.207 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:53:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:53:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 17 = '' 2023-12-02T02:53:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:53:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T02:53:19.009 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:53:19.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:53:19.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:53:19.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:53:19.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:53:19.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:53:19.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:53:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:53:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:53:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:53:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:53:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:53:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:53:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 17 = '' 2023-12-02T02:53:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:53:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1.6 2023-12-02T02:53:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:53:20.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:53:20.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:53:20.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:53:20.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:53:20.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:53:20.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:53:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:53:20.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=17 2023-12-02T02:53:20.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 17 = 17 2023-12-02T02:53:20.749 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:53:20.749 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2023-12-02T02:53:20.749 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:53:20.749 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:53:20.749 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:53:20.749 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap4 2023-12-02T02:53:20.749 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T02:53:20.749 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T02:53:20.749 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:53:21.749 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2023-12-02T02:53:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'test': 2023-12-02T02:53:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 512 MiB in 128 objects 2023-12-02T02:53:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T02:53:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 8 2023-12-02T02:53:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 10b949c95412 2023-12-02T02:53:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.10b949c95412 2023-12-02T02:53:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T02:53:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T02:53:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T02:53:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T02:53:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 02:50:03 2023 2023-12-02T02:53:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 02:51:39 2023 2023-12-02T02:53:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 02:50:03 2023 2023-12-02T02:53:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T02:53:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T02:53:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T02:53:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:53:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T02:53:21.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:53:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snap3-clean ']' 2023-12-02T02:53:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap3-clean 2023-12-02T02:53:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:53:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:53:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap3-clean 2023-12-02T02:53:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:53:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:53:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:53:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:53:21.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:53:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:53:22.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:53:22.889 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5931520021940626) > weight (0.25) 2023-12-02T02:53:22.889 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9953878661731942) > weight (0.25) 2023-12-02T02:53:22.889 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7016216467609058) > weight (0.25) 2023-12-02T02:53:22.889 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-12-02T02:53:22.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-12-02T02:53:22.890 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-12-02T02:53:22.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi057.stderr:2023-12-02T02:53:22.888+0000 7fa747c99640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 91880) UID: 1000 2023-12-02T02:53:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:53:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:53:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:53:23.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:53:23.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap4 2023-12-02T02:53:23.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:53:23.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:53:23.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap4 2023-12-02T02:53:23.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:53:23.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:53:23.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:53:23.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:53:23.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:53:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:53:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:53:24.891 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-12-02T02:53:24.891 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T02:53:24.891 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2023-12-02T02:53:24.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-12-02T02:53:24.891 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-12-02T02:53:24.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-12-02T02:53:24.894 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2023-12-02T02:53:25.054 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:53:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:53:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:53:25.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:53:25.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ clean_snap_name=snap4-clean 2023-12-02T02:53:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror test snap4-clean 2023-12-02T02:53:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:53:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:53:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:53:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap4-clean 2023-12-02T02:53:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2023-12-02T02:53:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:53:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror test 2023-12-02T02:53:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:53:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:53:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:53:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T02:53:26.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=9 2023-12-02T02:53:26.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T02:53:26.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-02T02:53:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 911360 bytes 2023-12-02T02:53:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 4934656 bytes 2023-12-02T02:53:26.773 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 67499008 bytes 2023-12-02T02:53:26.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 80366592 bytes 2023-12-02T02:53:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 108718080 bytes 2023-12-02T02:53:27.076 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 186189824 bytes 2023-12-02T02:53:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 255215616 bytes 2023-12-02T02:53:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 260428800 bytes 2023-12-02T02:53:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 282509312 bytes 2023-12-02T02:53:27.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 330764288 bytes 2023-12-02T02:53:27.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 370919424 bytes 2023-12-02T02:53:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 464550912 bytes 2023-12-02T02:53:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 505052160 bytes 2023-12-02T02:53:27.886 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 510383104 bytes 2023-12-02T02:53:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 603482112 bytes 2023-12-02T02:53:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 683856896 bytes 2023-12-02T02:53:28.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 690037760 bytes 2023-12-02T02:53:28.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 760487936 bytes 2023-12-02T02:53:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 785006592 bytes 2023-12-02T02:53:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 836088832 bytes 2023-12-02T02:53:28.731 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 889897984 bytes 2023-12-02T02:53:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 951051264 bytes 2023-12-02T02:53:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 973606912 bytes 2023-12-02T02:53:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 974240768 bytes 2023-12-02T02:53:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 987880448 bytes 2023-12-02T02:53:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1010776064 bytes 2023-12-02T02:53:29.276 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1060476928 bytes 2023-12-02T02:53:29.448 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1158288384 bytes 2023-12-02T02:53:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1175025664 bytes 2023-12-02T02:53:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1220940800 bytes 2023-12-02T02:53:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1246222336 bytes 2023-12-02T02:53:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1321247744 bytes 2023-12-02T02:53:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1375188992 bytes 2023-12-02T02:53:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1388210176 bytes 2023-12-02T02:53:30.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1399533568 bytes 2023-12-02T02:53:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1419823104 bytes 2023-12-02T02:53:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1444390912 bytes 2023-12-02T02:53:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1466063872 bytes 2023-12-02T02:53:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1536946176 bytes 2023-12-02T02:53:30.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1566729216 bytes 2023-12-02T02:53:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1645040640 bytes 2023-12-02T02:53:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1683640320 bytes 2023-12-02T02:53:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1758254080 bytes 2023-12-02T02:53:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1812139008 bytes 2023-12-02T02:53:31.492 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1925315584 bytes 2023-12-02T02:53:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 2001900544 bytes 2023-12-02T02:53:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 2061614080 bytes 2023-12-02T02:53:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2099499008 bytes 2023-12-02T02:53:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2154858496 bytes 2023-12-02T02:53:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2178013184 bytes 2023-12-02T02:53:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2208573440 bytes 2023-12-02T02:53:32.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2242560000 bytes 2023-12-02T02:53:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2294721536 bytes 2023-12-02T02:53:32.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2373705728 bytes 2023-12-02T02:53:32.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2400773120 bytes 2023-12-02T02:53:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2470952960 bytes 2023-12-02T02:53:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2495755264 bytes 2023-12-02T02:53:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2527499264 bytes 2023-12-02T02:53:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2592966656 bytes 2023-12-02T02:53:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2652748800 bytes 2023-12-02T02:53:33.307 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2697456640 bytes 2023-12-02T02:53:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2759568384 bytes 2023-12-02T02:53:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2776050688 bytes 2023-12-02T02:53:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2803840000 bytes 2023-12-02T02:53:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2865718272 bytes 2023-12-02T02:53:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2938627072 bytes 2023-12-02T02:53:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2970700800 bytes 2023-12-02T02:53:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 3015742464 bytes 2023-12-02T02:53:34.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 3120540672 bytes 2023-12-02T02:53:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 3142910976 bytes 2023-12-02T02:53:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 3157915648 bytes 2023-12-02T02:53:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 3187771392 bytes 2023-12-02T02:53:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 3213869056 bytes 2023-12-02T02:53:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3242011648 bytes 2023-12-02T02:53:34.836 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3311670272 bytes 2023-12-02T02:53:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3403603968 bytes 2023-12-02T02:53:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3435937792 bytes 2023-12-02T02:53:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3456885760 bytes 2023-12-02T02:53:35.345 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3546247168 bytes 2023-12-02T02:53:35.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T02:53:35.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T02:53:35.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T02:53:35.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:53:35.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-02T02:53:35.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:53:35.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:53:35.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:53:35.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:53:35.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:53:35.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:53:35.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:53:35.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:53:36.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T02:53:36.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:53:36.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:53:36.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:53:36.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:53:36.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:53:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:53:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T02:53:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T02:53:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:53:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:53:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:53:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T02:53:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ tee /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:53:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:test: 2023-12-02T02:53:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: global_id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:53:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: up+replaying 2023-12-02T02:53:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: replaying, {"bytes_per_second":17871906.13,"bytes_per_snapshot":536451584.0,"last_snapshot_bytes":536157184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701485596,"remote_snapshot_timestamp":1701485596,"replay_state":"idle"} 2023-12-02T02:53:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: service: mirror.2 on smithi057 2023-12-02T02:53:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02 02:53:31 2023-12-02T02:53:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: peer_sites: 2023-12-02T02:53:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: name: cluster2 2023-12-02T02:53:36.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: up+stopped 2023-12-02T02:53:36.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: local image is primary 2023-12-02T02:53:36.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02 02:53:31 2023-12-02T02:53:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:53:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T02:53:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ description: .*' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:53:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":17871906.13,"bytes_per_snapshot":536451584.0,"last_snapshot_bytes":536157184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701485596,"remote_snapshot_timestamp":1701485596,"replay_state":"idle"} 2023-12-02T02:53:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T02:53:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo up+replaying 2023-12-02T02:53:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^up+' 2023-12-02T02:53:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:up+replaying 2023-12-02T02:53:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:53:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.2 on smithi057 2023-12-02T02:53:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:53:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local 'last_update=2023-12-02 02:53:31' 2023-12-02T02:53:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-02 02:53:31' 2023-12-02T02:53:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:53:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:53:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:53:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:53:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local 'prev_last_update=2023-12-02 02:53:31' 2023-12-02T02:53:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-02T02:53:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror.mirror_status 2023-12-02T02:53:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:53:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror.mirror_status 2023-12-02T02:53:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-02T02:53:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T02:53:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local last_update state 2023-12-02T02:53:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-02T02:53:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ last_update='2023-12-02 02:53:31' 2023-12-02T02:53:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-02T02:53:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ state=up+replaying 2023-12-02T02:53:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo up+replaying 2023-12-02T02:53:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep up+replaying 2023-12-02T02:53:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:up+replaying 2023-12-02T02:53:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:53:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:53:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap5 2023-12-02T02:53:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror test snap5 2023-12-02T02:53:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:53:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:53:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:53:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap5 2023-12-02T02:53:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2023-12-02T02:54:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-02T02:54:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:54:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-02T02:54:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:54:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T02:54:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T02:54:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:54:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:54:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-02T02:54:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:54:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T02:54:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T02:54:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T02:54:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-02T02:54:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T02:54:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:54:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T02:54:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T02:54:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T02:54:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T02:54:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:54:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T02:54:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T02:54:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T02:54:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:54:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T02:54:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T02:54:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:54:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:54:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T02:54:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T02:54:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:54:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:54:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:54:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T02:54:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T02:54:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:54:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:54:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T02:54:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:54:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T02:54:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T02:54:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:54:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:54:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T02:54:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T02:54:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:54:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:54:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:54:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:54:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T02:54:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:54:06.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:54:06.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T02:54:06.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T02:54:06.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T02:54:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:54:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:54:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:54:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T02:54:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:54:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:54:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:54:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:54:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-02T02:54:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T02:54:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:54:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:54:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T02:54:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T02:54:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:54:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:54:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:54:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T02:54:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T02:54:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:54:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:54:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T02:54:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:54:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T02:54:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T02:54:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:54:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:54:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T02:54:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T02:54:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:54:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:54:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:54:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:54:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T02:54:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:54:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:54:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T02:54:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T02:54:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T02:54:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:54:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:54:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:54:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-02T02:54:06.603 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:54:06.603 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:54:06.603 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:54:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:54:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-02T02:54:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T02:54:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:54:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:54:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T02:54:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T02:54:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:54:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:54:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:54:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T02:54:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T02:54:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:54:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:54:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T02:54:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:54:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T02:54:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T02:54:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:54:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:54:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T02:54:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T02:54:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:54:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:54:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:54:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:54:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T02:54:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:54:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:54:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T02:54:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T02:54:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T02:54:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:54:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:54:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:54:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-02T02:54:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T02:54:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T02:54:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T02:54:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T02:54:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T02:54:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T02:54:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T02:54:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:54:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-02T02:54:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T02:54:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:54:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T02:54:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T02:54:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-02T02:54:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T02:54:06.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:54:06.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:06.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:06.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-test.status 2023-12-02T02:54:06.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-test.status 2023-12-02T02:54:06.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:54:06.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:54:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.status 2023-12-02T02:54:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.status 2023-12-02T02:54:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:54:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:54:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-02T02:54:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:54:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-02T02:54:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 21 2023-12-02T02:54:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:54:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:54:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:54:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-02T02:54:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:54:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T02:54:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=21 2023-12-02T02:54:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:54:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:54:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T02:54:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:54:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:54:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:54:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:54:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:54:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:54:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:54:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:54:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:54:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:54:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:54:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 21 = '' 2023-12-02T02:54:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:54:08.758 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T02:54:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:54:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:54:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:54:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:54:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:54:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:54:09.213 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:54:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:54:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:54:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:54:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:54:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 21 = '' 2023-12-02T02:54:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:54:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T02:54:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:54:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:54:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:54:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:54:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:54:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:54:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:54:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:54:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:54:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:54:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:54:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 21 = '' 2023-12-02T02:54:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:54:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1.6 2023-12-02T02:54:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:54:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:54:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:54:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:54:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:54:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:54:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=21 2023-12-02T02:54:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 21 = 21 2023-12-02T02:54:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:54:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2023-12-02T02:54:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:54:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap5 2023-12-02T02:54:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T02:54:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T02:54:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:54:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2023-12-02T02:54:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'test': 2023-12-02T02:54:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 512 MiB in 128 objects 2023-12-02T02:54:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T02:54:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 10 2023-12-02T02:54:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 10b949c95412 2023-12-02T02:54:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.10b949c95412 2023-12-02T02:54:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T02:54:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T02:54:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T02:54:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T02:54:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 02:50:03 2023 2023-12-02T02:54:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 02:53:22 2023 2023-12-02T02:54:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 02:50:03 2023 2023-12-02T02:54:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T02:54:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T02:54:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T02:54:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:54:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T02:54:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:54:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snap4-clean ']' 2023-12-02T02:54:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap4-clean 2023-12-02T02:54:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap4-clean 2023-12-02T02:54:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:54:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:54:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:54:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:54:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:54:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:54:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:54:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:54:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:54:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:54:14.558 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:54:14.558 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap5 2023-12-02T02:54:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap5 2023-12-02T02:54:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:54:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:54:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:54:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:54:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:54:15.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:54:15.125 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:54:15.955 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:54:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:54:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:54:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:54:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ clean_snap_name=snap5-clean 2023-12-02T02:54:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror test snap5-clean 2023-12-02T02:54:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:54:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap5-clean 2023-12-02T02:54:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2023-12-02T02:54:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:54:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror test 2023-12-02T02:54:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:54:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T02:54:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=9 2023-12-02T02:54:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T02:54:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-02T02:54:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 54764544 bytes 2023-12-02T02:54:17.708 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 77331456 bytes 2023-12-02T02:54:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 77975552 bytes 2023-12-02T02:54:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 128979968 bytes 2023-12-02T02:54:18.057 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 202339328 bytes 2023-12-02T02:54:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 253729792 bytes 2023-12-02T02:54:18.251 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 271636480 bytes 2023-12-02T02:54:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 279812096 bytes 2023-12-02T02:54:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 378782720 bytes 2023-12-02T02:54:18.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 414965760 bytes 2023-12-02T02:54:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 432882688 bytes 2023-12-02T02:54:18.783 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 482456576 bytes 2023-12-02T02:54:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 497783808 bytes 2023-12-02T02:54:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 603771904 bytes 2023-12-02T02:54:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 618452992 bytes 2023-12-02T02:54:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 641403904 bytes 2023-12-02T02:54:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 681979904 bytes 2023-12-02T02:54:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 699781120 bytes 2023-12-02T02:54:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 700397568 bytes 2023-12-02T02:54:19.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 718231552 bytes 2023-12-02T02:54:19.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 787121152 bytes 2023-12-02T02:54:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 821264384 bytes 2023-12-02T02:54:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 907866112 bytes 2023-12-02T02:54:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 977328128 bytes 2023-12-02T02:54:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 989025280 bytes 2023-12-02T02:54:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1083316224 bytes 2023-12-02T02:54:20.466 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1180099584 bytes 2023-12-02T02:54:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1202404352 bytes 2023-12-02T02:54:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1254740992 bytes 2023-12-02T02:54:20.789 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1295805440 bytes 2023-12-02T02:54:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1333644288 bytes 2023-12-02T02:54:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1390321664 bytes 2023-12-02T02:54:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1451025408 bytes 2023-12-02T02:54:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1482238976 bytes 2023-12-02T02:54:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1501173760 bytes 2023-12-02T02:54:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1507070976 bytes 2023-12-02T02:54:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1542296576 bytes 2023-12-02T02:54:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1551282176 bytes 2023-12-02T02:54:21.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1601774592 bytes 2023-12-02T02:54:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1629201408 bytes 2023-12-02T02:54:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1658643456 bytes 2023-12-02T02:54:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1710310400 bytes 2023-12-02T02:54:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1735987200 bytes 2023-12-02T02:54:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1760569344 bytes 2023-12-02T02:54:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1820965888 bytes 2023-12-02T02:54:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1838375936 bytes 2023-12-02T02:54:22.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1901905920 bytes 2023-12-02T02:54:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 1914039296 bytes 2023-12-02T02:54:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 1938964480 bytes 2023-12-02T02:54:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 1972067328 bytes 2023-12-02T02:54:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2048400384 bytes 2023-12-02T02:54:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2107619328 bytes 2023-12-02T02:54:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2124806144 bytes 2023-12-02T02:54:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2194569216 bytes 2023-12-02T02:54:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2249597952 bytes 2023-12-02T02:54:23.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2303965184 bytes 2023-12-02T02:54:23.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2330989568 bytes 2023-12-02T02:54:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2357045248 bytes 2023-12-02T02:54:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2424615936 bytes 2023-12-02T02:54:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2463039488 bytes 2023-12-02T02:54:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2499108864 bytes 2023-12-02T02:54:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2499733504 bytes 2023-12-02T02:54:24.198 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2532772864 bytes 2023-12-02T02:54:24.346 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2599663616 bytes 2023-12-02T02:54:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2648033280 bytes 2023-12-02T02:54:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2721914880 bytes 2023-12-02T02:54:24.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2761678848 bytes 2023-12-02T02:54:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2821433344 bytes 2023-12-02T02:54:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 2894860288 bytes 2023-12-02T02:54:25.172 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 2991086592 bytes 2023-12-02T02:54:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 3046768640 bytes 2023-12-02T02:54:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 3100624896 bytes 2023-12-02T02:54:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 3102496768 bytes 2023-12-02T02:54:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3130332160 bytes 2023-12-02T02:54:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3163227136 bytes 2023-12-02T02:54:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3241887744 bytes 2023-12-02T02:54:25.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3291719680 bytes 2023-12-02T02:54:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3316320256 bytes 2023-12-02T02:54:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3370188800 bytes 2023-12-02T02:54:26.333 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3464469504 bytes 2023-12-02T02:54:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T02:54:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T02:54:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T02:54:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:54:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-02T02:54:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:54:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:54:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:54:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:54:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:54:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:54:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T02:54:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:54:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:54:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:54:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:54:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T02:54:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T02:54:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:54:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:54:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:54:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T02:54:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ tee /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:54:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:test: 2023-12-02T02:54:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: global_id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:54:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: up+replaying 2023-12-02T02:54:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536451584.0,"last_snapshot_bytes":536157184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701485596,"remote_snapshot_timestamp":1701485596,"replay_state":"idle"} 2023-12-02T02:54:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: service: mirror.2 on smithi057 2023-12-02T02:54:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02 02:54:01 2023-12-02T02:54:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: peer_sites: 2023-12-02T02:54:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: name: cluster2 2023-12-02T02:54:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: up+stopped 2023-12-02T02:54:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: local image is primary 2023-12-02T02:54:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02 02:54:01 2023-12-02T02:54:27.470 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:54:27.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T02:54:27.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ description: .*' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:54:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536451584.0,"last_snapshot_bytes":536157184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701485596,"remote_snapshot_timestamp":1701485596,"replay_state":"idle"} 2023-12-02T02:54:27.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T02:54:27.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo up+replaying 2023-12-02T02:54:27.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^up+' 2023-12-02T02:54:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:up+replaying 2023-12-02T02:54:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:54:27.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.2 on smithi057 2023-12-02T02:54:27.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:54:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local 'last_update=2023-12-02 02:54:01' 2023-12-02T02:54:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-02 02:54:01' 2023-12-02T02:54:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:54:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:54:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local 'prev_last_update=2023-12-02 02:54:01' 2023-12-02T02:54:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-02T02:54:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror.mirror_status 2023-12-02T02:54:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:54:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror.mirror_status 2023-12-02T02:54:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-02T02:54:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T02:54:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local last_update state 2023-12-02T02:54:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-02T02:54:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ last_update='2023-12-02 02:54:01' 2023-12-02T02:54:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-02T02:54:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ state=up+replaying 2023-12-02T02:54:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo up+replaying 2023-12-02T02:54:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep up+replaying 2023-12-02T02:54:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:up+replaying 2023-12-02T02:54:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:54:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:54:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap6 2023-12-02T02:54:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror test snap6 2023-12-02T02:54:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:54:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap6 2023-12-02T02:54:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2023-12-02T02:54:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-02T02:54:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:54:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-02T02:54:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:54:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T02:54:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T02:54:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:54:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:54:53.895 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5147652421970067) > weight (0.25) 2023-12-02T02:54:53.895 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-02T02:54:53.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-02T02:54:53.896 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-02T02:54:53.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi057.stderr:2023-12-02T02:54:53.894+0000 7f39b4f25640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 91778) UID: 1000 2023-12-02T02:54:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-02T02:54:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:54:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T02:54:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T02:54:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T02:54:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-02T02:54:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T02:54:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T02:54:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:54:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T02:54:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T02:54:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T02:54:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:54:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T02:54:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T02:54:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T02:54:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:54:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T02:54:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T02:54:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:54:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:54:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T02:54:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T02:54:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:54:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:54:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:54:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T02:54:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T02:54:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:54:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:54:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T02:54:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:54:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T02:54:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T02:54:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:54:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:54:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T02:54:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T02:54:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:54:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:54:54.480 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:54:54.480 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:54:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T02:54:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:54:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:54:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T02:54:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T02:54:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T02:54:54.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:54:54.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:54:54.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:54:54.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T02:54:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:54:54.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:54:54.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:54:54.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:54:54.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-02T02:54:54.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T02:54:54.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:54:54.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:54:54.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T02:54:54.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T02:54:54.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:54:54.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:54:54.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:54:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T02:54:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T02:54:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:54:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:54:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T02:54:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:54:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T02:54:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T02:54:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:54:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:54:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T02:54:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T02:54:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:54:54.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:54:54.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:54:54.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:54:54.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T02:54:54.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:54:54.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:54:54.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T02:54:54.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T02:54:54.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T02:54:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:54:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:54:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:54:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-02T02:54:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:54:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:54:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:54:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:54:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-02T02:54:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T02:54:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:54:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:54:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T02:54:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T02:54:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:54:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:54:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:54:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T02:54:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T02:54:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:54:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:54:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T02:54:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:54:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T02:54:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T02:54:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:54:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:54:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T02:54:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T02:54:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:54:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:54:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:54:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:54:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T02:54:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:54:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:54:54.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T02:54:54.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T02:54:54.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T02:54:54.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:54:54.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:54:54.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:54:54.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-02T02:54:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T02:54:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T02:54:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T02:54:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T02:54:54.789 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T02:54:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T02:54:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T02:54:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:54:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-02T02:54:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T02:54:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:54:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T02:54:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T02:54:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-02T02:54:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T02:54:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:54:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-test.status 2023-12-02T02:54:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-test.status 2023-12-02T02:54:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:54:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:54:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.status 2023-12-02T02:54:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.status 2023-12-02T02:54:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:54:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:54:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-02T02:54:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:54:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:54.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:54.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-02T02:54:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 25 2023-12-02T02:54:56.490 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:54:56.490 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:54:56.490 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:56.490 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:56.490 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:54:56.491 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-02T02:54:56.491 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:54:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T02:54:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=25 2023-12-02T02:54:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:54:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:54:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T02:54:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:54:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:54:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:54:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:54:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:54:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:54:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:54:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:54:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:54:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:54:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:54:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 25 = '' 2023-12-02T02:54:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:54:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T02:54:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:54:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:54:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:54:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:54:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:54:57.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:54:57.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:54:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:54:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:54:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:54:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:54:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 25 = '' 2023-12-02T02:54:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:54:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T02:54:57.897 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-02T02:54:57.897 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T02:54:57.897 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-02T02:54:57.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-02T02:54:57.898 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-02T02:54:57.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-02T02:54:57.901 INFO:tasks.rbd_mirror_thrash:waiting for 105 secs before thrashing 2023-12-02T02:54:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:54:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:54:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:54:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:54:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:54:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:54:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:54:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:54:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:54:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:54:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:54:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 25 = '' 2023-12-02T02:54:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:54:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1.6 2023-12-02T02:54:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:54:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:54:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:54:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:54:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:54:59.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:54:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=25 2023-12-02T02:54:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 25 = 25 2023-12-02T02:54:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:54:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2023-12-02T02:54:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:54:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:54:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:54:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap6 2023-12-02T02:54:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T02:54:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T02:54:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:55:00.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2023-12-02T02:55:00.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'test': 2023-12-02T02:55:00.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 512 MiB in 128 objects 2023-12-02T02:55:00.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T02:55:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 12 2023-12-02T02:55:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 10b949c95412 2023-12-02T02:55:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.10b949c95412 2023-12-02T02:55:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T02:55:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T02:55:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T02:55:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T02:55:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 02:50:03 2023 2023-12-02T02:55:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 02:53:22 2023 2023-12-02T02:55:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 02:50:03 2023 2023-12-02T02:55:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T02:55:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T02:55:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T02:55:00.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:55:00.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T02:55:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:55:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snap5-clean ']' 2023-12-02T02:55:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap5-clean 2023-12-02T02:55:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:55:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:55:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap5-clean 2023-12-02T02:55:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:55:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:55:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:55:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:55:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:55:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:55:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:55:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:55:02.273 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:55:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:55:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:55:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap6 2023-12-02T02:55:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:55:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:55:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap6 2023-12-02T02:55:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:55:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:55:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:55:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:55:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:55:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:55:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:55:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:55:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:55:04.318 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:55:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:55:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ clean_snap_name=snap6-clean 2023-12-02T02:55:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror test snap6-clean 2023-12-02T02:55:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:55:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:55:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:55:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap6-clean 2023-12-02T02:55:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2023-12-02T02:55:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:55:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror test 2023-12-02T02:55:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:55:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:55:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:55:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T02:55:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=14 2023-12-02T02:55:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T02:55:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-02T02:55:05.693 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 100133888 bytes 2023-12-02T02:55:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 117657600 bytes 2023-12-02T02:55:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 118260736 bytes 2023-12-02T02:55:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 173305856 bytes 2023-12-02T02:55:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 223719424 bytes 2023-12-02T02:55:06.185 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 246371328 bytes 2023-12-02T02:55:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 312700928 bytes 2023-12-02T02:55:06.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 330448896 bytes 2023-12-02T02:55:06.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 339223552 bytes 2023-12-02T02:55:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 426629120 bytes 2023-12-02T02:55:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 470330368 bytes 2023-12-02T02:55:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 495387648 bytes 2023-12-02T02:55:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 521011200 bytes 2023-12-02T02:55:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 522739712 bytes 2023-12-02T02:55:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 584701952 bytes 2023-12-02T02:55:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 644287488 bytes 2023-12-02T02:55:07.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 680949760 bytes 2023-12-02T02:55:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 701683712 bytes 2023-12-02T02:55:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 724318208 bytes 2023-12-02T02:55:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 771411968 bytes 2023-12-02T02:55:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 788251648 bytes 2023-12-02T02:55:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 850106368 bytes 2023-12-02T02:55:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 898928640 bytes 2023-12-02T02:55:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 930252800 bytes 2023-12-02T02:55:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1018457088 bytes 2023-12-02T02:55:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1036018688 bytes 2023-12-02T02:55:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1126016000 bytes 2023-12-02T02:55:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1140683776 bytes 2023-12-02T02:55:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1168713728 bytes 2023-12-02T02:55:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1227534336 bytes 2023-12-02T02:55:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1266359296 bytes 2023-12-02T02:55:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1367294976 bytes 2023-12-02T02:55:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1385041920 bytes 2023-12-02T02:55:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1444179968 bytes 2023-12-02T02:55:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1480264704 bytes 2023-12-02T02:55:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1538113536 bytes 2023-12-02T02:55:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1580816384 bytes 2023-12-02T02:55:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1635275776 bytes 2023-12-02T02:55:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1644417024 bytes 2023-12-02T02:55:09.897 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3747085477021721) > weight (0.25) 2023-12-02T02:55:09.897 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7101107524659553) > weight (0.25) 2023-12-02T02:55:09.897 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-02T02:55:09.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-02T02:55:09.898 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-02T02:55:09.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr:2023-12-02T02:55:09.896+0000 7fe3a6718640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 91657) UID: 1000 2023-12-02T02:55:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1681129472 bytes 2023-12-02T02:55:10.067 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1760335872 bytes 2023-12-02T02:55:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1877268480 bytes 2023-12-02T02:55:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1948967936 bytes 2023-12-02T02:55:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 2003173376 bytes 2023-12-02T02:55:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 2084189184 bytes 2023-12-02T02:55:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 2191022080 bytes 2023-12-02T02:55:10.960 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 2195045376 bytes 2023-12-02T02:55:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2262184960 bytes 2023-12-02T02:55:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2276512768 bytes 2023-12-02T02:55:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2313455616 bytes 2023-12-02T02:55:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2384876544 bytes 2023-12-02T02:55:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2424822784 bytes 2023-12-02T02:55:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2462315520 bytes 2023-12-02T02:55:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2548878336 bytes 2023-12-02T02:55:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2583351296 bytes 2023-12-02T02:55:12.040 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2632120320 bytes 2023-12-02T02:55:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2684938240 bytes 2023-12-02T02:55:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2775757824 bytes 2023-12-02T02:55:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2819599360 bytes 2023-12-02T02:55:12.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2848975872 bytes 2023-12-02T02:55:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2858803200 bytes 2023-12-02T02:55:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2890269696 bytes 2023-12-02T02:55:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2912536576 bytes 2023-12-02T02:55:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2955464704 bytes 2023-12-02T02:55:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 3036688384 bytes 2023-12-02T02:55:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 3099929600 bytes 2023-12-02T02:55:13.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 3207658496 bytes 2023-12-02T02:55:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 3228161024 bytes 2023-12-02T02:55:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 3255234560 bytes 2023-12-02T02:55:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 3286174720 bytes 2023-12-02T02:55:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 3349300224 bytes 2023-12-02T02:55:13.986 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 3404500992 bytes 2023-12-02T02:55:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 3457081344 bytes 2023-12-02T02:55:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3471197184 bytes 2023-12-02T02:55:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3480466432 bytes 2023-12-02T02:55:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3565991936 bytes 2023-12-02T02:55:14.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3586046976 bytes 2023-12-02T02:55:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3647931392 bytes 2023-12-02T02:55:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3683160064 bytes 2023-12-02T02:55:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3730522112 bytes 2023-12-02T02:55:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3766321152 bytes 2023-12-02T02:55:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8200 IOs, 3796243456 bytes 2023-12-02T02:55:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8300 IOs, 3831769088 bytes 2023-12-02T02:55:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8400 IOs, 3860728832 bytes 2023-12-02T02:55:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8500 IOs, 3885376512 bytes 2023-12-02T02:55:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8600 IOs, 3949794304 bytes 2023-12-02T02:55:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8700 IOs, 4032962560 bytes 2023-12-02T02:55:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8800 IOs, 4104518656 bytes 2023-12-02T02:55:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8900 IOs, 4160221184 bytes 2023-12-02T02:55:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9000 IOs, 4193224704 bytes 2023-12-02T02:55:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9100 IOs, 4241875968 bytes 2023-12-02T02:55:16.322 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9200 IOs, 4311169024 bytes 2023-12-02T02:55:16.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9300 IOs, 4383363072 bytes 2023-12-02T02:55:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9400 IOs, 4410178560 bytes 2023-12-02T02:55:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9500 IOs, 4440337408 bytes 2023-12-02T02:55:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9600 IOs, 4477902848 bytes 2023-12-02T02:55:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9700 IOs, 4515363840 bytes 2023-12-02T02:55:16.898 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-02T02:55:16.898 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T02:55:16.899 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-02T02:55:16.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-02T02:55:16.899 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-02T02:55:16.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-02T02:55:16.902 INFO:tasks.rbd_mirror_thrash:waiting for 107 secs before thrashing 2023-12-02T02:55:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9800 IOs, 4555201536 bytes 2023-12-02T02:55:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9900 IOs, 4576254976 bytes 2023-12-02T02:55:17.235 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10000 IOs, 4646003712 bytes 2023-12-02T02:55:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10100 IOs, 4686614528 bytes 2023-12-02T02:55:17.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10200 IOs, 4716044288 bytes 2023-12-02T02:55:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10300 IOs, 4795474944 bytes 2023-12-02T02:55:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10400 IOs, 4855916544 bytes 2023-12-02T02:55:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10500 IOs, 4914156544 bytes 2023-12-02T02:55:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10600 IOs, 4993836032 bytes 2023-12-02T02:55:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10700 IOs, 5073992704 bytes 2023-12-02T02:55:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10800 IOs, 5157417984 bytes 2023-12-02T02:55:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10900 IOs, 5218102272 bytes 2023-12-02T02:55:18.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11000 IOs, 5254821888 bytes 2023-12-02T02:55:18.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11100 IOs, 5327540224 bytes 2023-12-02T02:55:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11200 IOs, 5384548352 bytes 2023-12-02T02:55:18.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11300 IOs, 5431659520 bytes 2023-12-02T02:55:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11400 IOs, 5462982656 bytes 2023-12-02T02:55:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11500 IOs, 5502395392 bytes 2023-12-02T02:55:19.341 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11600 IOs, 5553343488 bytes 2023-12-02T02:55:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T02:55:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T02:55:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T02:55:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:55:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-02T02:55:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:55:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:55:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:55:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:55:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:55:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:55:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:55:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:55:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T02:55:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:55:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:55:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:55:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:55:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:55:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:55:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T02:55:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T02:55:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:55:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:55:20.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:55:20.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T02:55:20.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ tee /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:55:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:test: 2023-12-02T02:55:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: global_id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:55:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: down+unknown 2023-12-02T02:55:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: status not found 2023-12-02T02:55:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02T02:55:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: peer_sites: 2023-12-02T02:55:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: name: cluster2 2023-12-02T02:55:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: up+stopped 2023-12-02T02:55:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: local image is primary 2023-12-02T02:55:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02 02:55:01 2023-12-02T02:55:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:55:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 1 2023-12-02T02:55:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:55:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 2 2023-12-02T02:55:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T02:55:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:55:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:55:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:55:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:55:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:55:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:55:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T02:55:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T02:55:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:55:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:55:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:55:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T02:55:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ tee /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:55:22.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:test: 2023-12-02T02:55:22.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: global_id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:55:22.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: down+unknown 2023-12-02T02:55:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: status not found 2023-12-02T02:55:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02T02:55:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: peer_sites: 2023-12-02T02:55:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: name: cluster2 2023-12-02T02:55:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: up+stopped 2023-12-02T02:55:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: local image is primary 2023-12-02T02:55:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02 02:55:01 2023-12-02T02:55:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:55:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 1 2023-12-02T02:55:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:55:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 4 2023-12-02T02:55:26.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T02:55:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:55:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:55:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:55:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:55:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:55:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:55:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T02:55:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T02:55:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:55:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:55:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:55:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T02:55:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ tee /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:55:26.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:test: 2023-12-02T02:55:26.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: global_id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:55:26.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: down+unknown 2023-12-02T02:55:26.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: status not found 2023-12-02T02:55:26.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02T02:55:26.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: peer_sites: 2023-12-02T02:55:26.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: name: cluster2 2023-12-02T02:55:26.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: up+stopped 2023-12-02T02:55:26.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: local image is primary 2023-12-02T02:55:26.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02 02:55:01 2023-12-02T02:55:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:55:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 1 2023-12-02T02:55:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:55:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 8 2023-12-02T02:55:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T02:55:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:55:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:55:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:55:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:55:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:55:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:55:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T02:55:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T02:55:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:55:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:55:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:55:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T02:55:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ tee /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:55:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:test: 2023-12-02T02:55:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: global_id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:55:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: down+unknown 2023-12-02T02:55:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: status not found 2023-12-02T02:55:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02T02:55:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: peer_sites: 2023-12-02T02:55:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: name: cluster2 2023-12-02T02:55:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: up+stopped 2023-12-02T02:55:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: local image is primary 2023-12-02T02:55:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02 02:55:31 2023-12-02T02:55:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:55:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 1 2023-12-02T02:55:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:55:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 8 2023-12-02T02:55:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T02:55:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:55:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:55:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:55:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:55:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:55:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:55:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T02:55:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T02:55:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:55:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:55:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:55:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T02:55:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ tee /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:55:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:test: 2023-12-02T02:55:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: global_id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:55:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: down+unknown 2023-12-02T02:55:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: status not found 2023-12-02T02:55:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02T02:55:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: peer_sites: 2023-12-02T02:55:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: name: cluster2 2023-12-02T02:55:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: up+stopped 2023-12-02T02:55:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: local image is primary 2023-12-02T02:55:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02 02:55:31 2023-12-02T02:55:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:55:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 1 2023-12-02T02:55:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:55:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 8 2023-12-02T02:55:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T02:55:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:55:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:55:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:55:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:55:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:55:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:55:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T02:55:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T02:55:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:55:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:55:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:55:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T02:55:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ tee /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:55:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:test: 2023-12-02T02:55:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: global_id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:55:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: up+replaying 2023-12-02T02:55:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701485695,"remote_snapshot_timestamp":1701485695,"replay_state":"idle"} 2023-12-02T02:55:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: service: mirror.3 on smithi057 2023-12-02T02:55:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02 02:55:43 2023-12-02T02:55:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: peer_sites: 2023-12-02T02:55:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: name: cluster2 2023-12-02T02:55:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: up+stopped 2023-12-02T02:55:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: local image is primary 2023-12-02T02:55:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02 02:55:31 2023-12-02T02:55:50.814 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:55:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T02:55:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ description: .*' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:55:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701485695,"remote_snapshot_timestamp":1701485695,"replay_state":"idle"} 2023-12-02T02:55:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T02:55:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo up+replaying 2023-12-02T02:55:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^up+' 2023-12-02T02:55:50.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:up+replaying 2023-12-02T02:55:50.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:55:50.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.3 on smithi057 2023-12-02T02:55:50.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:55:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local 'last_update=2023-12-02 02:55:43' 2023-12-02T02:55:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-02 02:55:43' 2023-12-02T02:55:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:55:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:55:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:55:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:55:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local 'prev_last_update=2023-12-02 02:55:43' 2023-12-02T02:55:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-02T02:55:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror.mirror_status 2023-12-02T02:55:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:55:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror.mirror_status 2023-12-02T02:55:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-02T02:55:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T02:55:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local last_update state 2023-12-02T02:55:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-02T02:55:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ last_update='2023-12-02 02:55:43' 2023-12-02T02:55:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-02T02:55:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ state=up+replaying 2023-12-02T02:55:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo up+replaying 2023-12-02T02:55:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep up+replaying 2023-12-02T02:55:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:up+replaying 2023-12-02T02:55:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:55:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:55:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap7 2023-12-02T02:55:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror test snap7 2023-12-02T02:55:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:55:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:55:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:55:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap7 2023-12-02T02:55:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2023-12-02T02:55:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-02T02:55:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:55:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:55:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:55:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-02T02:55:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:55:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:55:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:55:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T02:55:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T02:55:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:55:51.466 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:55:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-02T02:55:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:55:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:55:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:55:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T02:55:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T02:55:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T02:55:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-02T02:55:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T02:55:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T02:55:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:55:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T02:55:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T02:55:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T02:55:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:55:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T02:55:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T02:55:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T02:55:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:55:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T02:55:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T02:55:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:55:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:55:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T02:55:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T02:55:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:55:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:55:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:55:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T02:55:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T02:55:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:55:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:55:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T02:55:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:55:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T02:55:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T02:55:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:55:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:55:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T02:55:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T02:55:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:55:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:55:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:55:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:55:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T02:55:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:55:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:55:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T02:55:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T02:55:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T02:55:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:55:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:55:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:55:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T02:55:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:55:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:55:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:55:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:55:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-02T02:55:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T02:55:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:55:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:55:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T02:55:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T02:55:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:55:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:55:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:55:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T02:55:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T02:55:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:55:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:55:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T02:55:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:55:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T02:55:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T02:55:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:55:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:55:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T02:55:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T02:55:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:55:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:55:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:55:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:55:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T02:55:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:55:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:55:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T02:55:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T02:55:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T02:55:52.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:55:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:55:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:55:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-02T02:55:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:55:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:55:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:55:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:55:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-02T02:55:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T02:55:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:55:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:55:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T02:55:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T02:55:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:55:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:55:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:55:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T02:55:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T02:55:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:55:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:55:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T02:55:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:55:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T02:55:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T02:55:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:55:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:55:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T02:55:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T02:55:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:55:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:55:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:55:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:55:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T02:55:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:55:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:55:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T02:55:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T02:55:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T02:55:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:55:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:55:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:55:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-02T02:55:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:55:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:55:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:55:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:55:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-02T02:55:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T02:55:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:55:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:55:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T02:55:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T02:55:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:55:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:55:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:55:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T02:55:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T02:55:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:55:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:55:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T02:55:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T02:55:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T02:55:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T02:55:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:55:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:55:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T02:55:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T02:55:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:55:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:55:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:55:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T02:55:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T02:55:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:55:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:55:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T02:55:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T02:55:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T02:55:52.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T02:55:52.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T02:55:52.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T02:55:52.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-02T02:55:52.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T02:55:52.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T02:55:52.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T02:55:52.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T02:55:52.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T02:55:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T02:55:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T02:55:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:55:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-02T02:55:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T02:55:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:55:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:55:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:55:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T02:55:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T02:55:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-02T02:55:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T02:55:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:55:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:55:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:55:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-test.status 2023-12-02T02:55:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-test.status 2023-12-02T02:55:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:55:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:55:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.status 2023-12-02T02:55:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.status 2023-12-02T02:55:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:55:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:55:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-02T02:55:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:55:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:55:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:55:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-02T02:55:54.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 29 2023-12-02T02:55:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:55:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:55:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:55:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:55:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:55:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:55:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-02T02:55:54.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T02:55:54.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=29 2023-12-02T02:55:54.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:55:54.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:55:54.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T02:55:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:55:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:55:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:55:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:55:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:55:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:55:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:55:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:55:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:55:54.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:55:54.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:55:54.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:55:54.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:55:54.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 29 = '' 2023-12-02T02:55:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:55:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T02:55:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:55:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:55:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:55:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:55:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:55:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:55:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:55:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:55:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:55:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:55:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:55:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:55:55.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:55:55.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 29 = '' 2023-12-02T02:55:55.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:55:55.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T02:55:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:55:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:55:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:55:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:55:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:55:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:55:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:55:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:55:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:55:56.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:55:56.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:55:56.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:55:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:55:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 29 = '' 2023-12-02T02:55:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:55:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1.6 2023-12-02T02:55:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:55:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:55:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:55:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:55:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:55:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:55:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:55:57.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:55:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=29 2023-12-02T02:55:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 29 = 29 2023-12-02T02:55:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:55:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2023-12-02T02:55:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:55:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:55:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:55:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap7 2023-12-02T02:55:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T02:55:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T02:55:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:55:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2023-12-02T02:55:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'test': 2023-12-02T02:55:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 512 MiB in 128 objects 2023-12-02T02:55:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T02:55:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 14 2023-12-02T02:55:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 10b949c95412 2023-12-02T02:55:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.10b949c95412 2023-12-02T02:55:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T02:55:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T02:55:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T02:55:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T02:55:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 02:50:03 2023 2023-12-02T02:55:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 02:55:01 2023 2023-12-02T02:55:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 02:50:03 2023 2023-12-02T02:55:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T02:55:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T02:55:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T02:55:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:55:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T02:55:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:55:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snap6-clean ']' 2023-12-02T02:55:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap6-clean 2023-12-02T02:55:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:55:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:55:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap6-clean 2023-12-02T02:55:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:55:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:55:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:55:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:55:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:55:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:55:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:56:00.237 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:56:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:56:00.492 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:56:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:56:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap7 2023-12-02T02:56:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:56:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:56:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap7 2023-12-02T02:56:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:56:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:56:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:56:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:56:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:56:01.181 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:56:01.190 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:56:01.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:56:02.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:56:02.246 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:56:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:56:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ clean_snap_name=snap7-clean 2023-12-02T02:56:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror test snap7-clean 2023-12-02T02:56:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:56:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:56:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:56:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap7-clean 2023-12-02T02:56:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2023-12-02T02:56:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:56:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror test 2023-12-02T02:56:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:56:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:56:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:56:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T02:56:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=6 2023-12-02T02:56:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T02:56:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-02T02:56:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 67830784 bytes 2023-12-02T02:56:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 179377152 bytes 2023-12-02T02:56:04.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 268081152 bytes 2023-12-02T02:56:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 307709952 bytes 2023-12-02T02:56:04.322 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 352954368 bytes 2023-12-02T02:56:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 385442816 bytes 2023-12-02T02:56:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 435848192 bytes 2023-12-02T02:56:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 494596096 bytes 2023-12-02T02:56:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 540135424 bytes 2023-12-02T02:56:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 550158336 bytes 2023-12-02T02:56:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 617442304 bytes 2023-12-02T02:56:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 635854848 bytes 2023-12-02T02:56:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 676806656 bytes 2023-12-02T02:56:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 715200512 bytes 2023-12-02T02:56:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 734374912 bytes 2023-12-02T02:56:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 807966720 bytes 2023-12-02T02:56:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 836240384 bytes 2023-12-02T02:56:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 933447680 bytes 2023-12-02T02:56:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 1009771520 bytes 2023-12-02T02:56:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 1013866496 bytes 2023-12-02T02:56:06.137 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 1041646592 bytes 2023-12-02T02:56:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 1042271232 bytes 2023-12-02T02:56:06.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 1082072064 bytes 2023-12-02T02:56:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1132115968 bytes 2023-12-02T02:56:06.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1181709312 bytes 2023-12-02T02:56:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1206771712 bytes 2023-12-02T02:56:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1237650432 bytes 2023-12-02T02:56:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1244514304 bytes 2023-12-02T02:56:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1281028096 bytes 2023-12-02T02:56:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1305338880 bytes 2023-12-02T02:56:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1331184640 bytes 2023-12-02T02:56:07.227 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1372428288 bytes 2023-12-02T02:56:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1412325376 bytes 2023-12-02T02:56:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1428540416 bytes 2023-12-02T02:56:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1452492800 bytes 2023-12-02T02:56:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1465742336 bytes 2023-12-02T02:56:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1496385536 bytes 2023-12-02T02:56:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1542546432 bytes 2023-12-02T02:56:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1593592832 bytes 2023-12-02T02:56:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1640056832 bytes 2023-12-02T02:56:08.158 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1656423424 bytes 2023-12-02T02:56:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1712132096 bytes 2023-12-02T02:56:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1747962880 bytes 2023-12-02T02:56:08.466 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1748571136 bytes 2023-12-02T02:56:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1779231744 bytes 2023-12-02T02:56:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1811431424 bytes 2023-12-02T02:56:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1851628544 bytes 2023-12-02T02:56:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 1895491584 bytes 2023-12-02T02:56:09.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 1965665280 bytes 2023-12-02T02:56:09.161 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2010250240 bytes 2023-12-02T02:56:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2053840896 bytes 2023-12-02T02:56:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2136809472 bytes 2023-12-02T02:56:09.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T02:56:09.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T02:56:09.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T02:56:09.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:56:09.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-02T02:56:09.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:56:09.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:56:09.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:56:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:56:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:56:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:56:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:56:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:56:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T02:56:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:56:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:56:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:56:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:56:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:56:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:56:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T02:56:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T02:56:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:56:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:56:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:56:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T02:56:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ tee /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:56:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:test: 2023-12-02T02:56:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: global_id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:56:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: up+replaying 2023-12-02T02:56:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: replaying, {"bytes_per_second":29060534.86,"bytes_per_snapshot":535205888.0,"last_snapshot_bytes":535205888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701485753,"remote_snapshot_timestamp":1701485753,"replay_state":"idle"} 2023-12-02T02:56:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: service: mirror.3 on smithi057 2023-12-02T02:56:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02 02:56:01 2023-12-02T02:56:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: peer_sites: 2023-12-02T02:56:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: name: cluster2 2023-12-02T02:56:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: up+stopped 2023-12-02T02:56:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: local image is primary 2023-12-02T02:56:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02 02:56:01 2023-12-02T02:56:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:56:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T02:56:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ description: .*' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:56:10.505 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":29060534.86,"bytes_per_snapshot":535205888.0,"last_snapshot_bytes":535205888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701485753,"remote_snapshot_timestamp":1701485753,"replay_state":"idle"} 2023-12-02T02:56:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T02:56:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo up+replaying 2023-12-02T02:56:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^up+' 2023-12-02T02:56:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:up+replaying 2023-12-02T02:56:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:56:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.3 on smithi057 2023-12-02T02:56:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:56:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local 'last_update=2023-12-02 02:56:01' 2023-12-02T02:56:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-02 02:56:01' 2023-12-02T02:56:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:56:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:56:10.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:56:10.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:56:10.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local 'prev_last_update=2023-12-02 02:56:01' 2023-12-02T02:56:10.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-02T02:56:10.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror.mirror_status 2023-12-02T02:56:10.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:56:10.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror.mirror_status 2023-12-02T02:56:10.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-02T02:56:10.549 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T02:56:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local last_update state 2023-12-02T02:56:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-02T02:56:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ last_update='2023-12-02 02:56:01' 2023-12-02T02:56:10.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-02T02:56:10.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ state=up+replaying 2023-12-02T02:56:10.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo up+replaying 2023-12-02T02:56:10.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep up+replaying 2023-12-02T02:56:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:up+replaying 2023-12-02T02:56:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:56:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:56:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap8 2023-12-02T02:56:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror test snap8 2023-12-02T02:56:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:56:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:56:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:56:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap8 2023-12-02T02:56:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2023-12-02T02:56:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-02T02:56:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:56:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:56:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:56:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-02T02:56:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:56:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:56:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:56:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T02:56:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T02:56:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:56:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:56:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-02T02:56:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:56:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:56:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:56:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T02:56:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T02:56:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T02:56:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-02T02:56:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T02:56:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T02:56:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:56:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T02:56:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T02:56:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T02:56:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:56:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T02:56:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T02:56:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T02:56:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:56:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T02:56:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T02:56:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:56:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:56:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T02:56:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T02:56:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:56:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:56:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:56:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T02:56:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T02:56:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:56:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:56:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T02:56:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:56:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T02:56:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T02:56:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:56:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:56:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T02:56:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T02:56:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:56:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:56:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:56:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:56:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T02:56:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:56:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:56:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T02:56:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T02:56:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T02:56:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:56:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:56:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:56:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T02:56:40.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:56:40.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:56:40.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:56:40.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:56:40.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-02T02:56:40.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T02:56:40.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:56:40.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:56:40.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T02:56:40.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T02:56:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:56:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:56:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:56:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T02:56:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T02:56:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:56:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:56:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T02:56:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:56:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T02:56:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T02:56:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:56:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:56:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T02:56:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T02:56:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:56:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:56:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:56:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:56:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T02:56:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:56:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:56:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T02:56:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T02:56:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T02:56:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:56:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:56:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:56:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-02T02:56:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:56:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:56:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:56:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:56:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-02T02:56:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T02:56:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:56:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:56:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T02:56:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T02:56:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:56:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:56:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:56:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T02:56:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T02:56:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:56:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:56:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T02:56:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:56:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T02:56:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T02:56:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:56:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:56:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T02:56:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T02:56:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:56:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:56:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:56:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:56:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T02:56:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:56:40.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:56:40.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T02:56:40.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T02:56:40.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T02:56:40.692 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:56:40.692 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:56:40.692 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:56:40.692 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-02T02:56:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:56:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:56:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:56:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:56:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-02T02:56:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T02:56:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:56:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:56:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T02:56:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T02:56:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:56:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:56:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:56:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T02:56:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T02:56:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:56:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:56:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T02:56:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T02:56:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T02:56:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T02:56:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:56:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:56:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T02:56:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T02:56:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:56:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:56:40.774 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:56:40.774 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T02:56:40.774 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T02:56:40.774 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:56:40.774 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:56:40.774 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T02:56:40.774 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T02:56:40.774 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T02:56:40.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T02:56:40.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T02:56:40.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T02:56:40.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-02T02:56:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T02:56:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T02:56:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T02:56:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T02:56:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T02:56:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T02:56:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T02:56:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:56:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-02T02:56:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T02:56:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:56:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:56:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:56:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T02:56:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T02:56:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-02T02:56:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T02:56:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:56:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:56:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:56:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-test.status 2023-12-02T02:56:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-test.status 2023-12-02T02:56:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:56:40.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:56:40.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.status 2023-12-02T02:56:40.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.status 2023-12-02T02:56:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:56:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:56:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-02T02:56:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:56:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:56:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:56:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-02T02:56:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 33 2023-12-02T02:56:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:56:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:56:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:56:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:56:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:56:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-02T02:56:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:56:42.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T02:56:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=33 2023-12-02T02:56:42.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:56:42.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:56:42.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T02:56:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:56:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:56:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:56:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:56:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:56:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:56:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:56:42.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:56:42.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:56:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:56:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:56:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:56:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:56:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 33 = '' 2023-12-02T02:56:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:56:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T02:56:42.901 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-02T02:56:42.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-02T02:56:42.902 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-02T02:56:42.902 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-02T02:56:42.902 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T02:56:42.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr:2023-12-02T02:56:42.900+0000 7fcab8e4a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 91751) UID: 1000 2023-12-02T02:56:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:56:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:56:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:56:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:56:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:56:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:56:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:56:43.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:56:43.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:56:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:56:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:56:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:56:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:56:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 33 = '' 2023-12-02T02:56:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:56:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T02:56:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:56:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:56:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:56:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:56:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:56:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:56:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:56:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:56:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:56:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:56:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:56:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:56:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:56:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 33 = '' 2023-12-02T02:56:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:56:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1.6 2023-12-02T02:56:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:56:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:56:45.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:56:45.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:56:45.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:56:45.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:56:45.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:56:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:56:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=33 2023-12-02T02:56:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 33 = 33 2023-12-02T02:56:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:56:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2023-12-02T02:56:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:56:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:56:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:56:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap8 2023-12-02T02:56:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T02:56:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T02:56:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:56:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2023-12-02T02:56:46.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'test': 2023-12-02T02:56:46.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 512 MiB in 128 objects 2023-12-02T02:56:46.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T02:56:46.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 16 2023-12-02T02:56:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 10b949c95412 2023-12-02T02:56:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.10b949c95412 2023-12-02T02:56:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T02:56:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T02:56:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T02:56:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T02:56:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 02:50:03 2023 2023-12-02T02:56:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 02:55:01 2023 2023-12-02T02:56:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 02:50:03 2023 2023-12-02T02:56:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T02:56:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T02:56:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T02:56:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:56:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T02:56:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:56:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snap7-clean ']' 2023-12-02T02:56:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap7-clean 2023-12-02T02:56:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:56:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:56:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap7-clean 2023-12-02T02:56:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:56:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:56:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:56:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:56:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:56:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:56:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:56:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:56:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:56:48.442 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:56:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:56:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap8 2023-12-02T02:56:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:56:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:56:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap8 2023-12-02T02:56:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:56:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:56:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:56:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:56:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:56:49.004 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-02T02:56:49.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-02T02:56:49.004 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-02T02:56:49.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-02T02:56:49.007 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2023-12-02T02:56:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:56:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:56:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:56:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:56:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:56:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:56:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ clean_snap_name=snap8-clean 2023-12-02T02:56:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror test snap8-clean 2023-12-02T02:56:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:56:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:56:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:56:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap8-clean 2023-12-02T02:56:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2023-12-02T02:56:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:56:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror test 2023-12-02T02:56:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:56:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:56:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:56:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T02:56:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=11 2023-12-02T02:56:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T02:56:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-02T02:56:50.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 22999040 bytes 2023-12-02T02:56:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 58622976 bytes 2023-12-02T02:56:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 83453952 bytes 2023-12-02T02:56:50.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 92788736 bytes 2023-12-02T02:56:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 141355008 bytes 2023-12-02T02:56:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 179975168 bytes 2023-12-02T02:56:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 230287360 bytes 2023-12-02T02:56:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 239120384 bytes 2023-12-02T02:56:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 270149632 bytes 2023-12-02T02:56:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 310923264 bytes 2023-12-02T02:56:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 359966720 bytes 2023-12-02T02:56:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 376572928 bytes 2023-12-02T02:56:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 390769664 bytes 2023-12-02T02:56:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 420815872 bytes 2023-12-02T02:56:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 469535744 bytes 2023-12-02T02:56:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 474347520 bytes 2023-12-02T02:56:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 493270016 bytes 2023-12-02T02:56:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 493935616 bytes 2023-12-02T02:56:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 509316096 bytes 2023-12-02T02:56:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 641241088 bytes 2023-12-02T02:56:52.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 675963904 bytes 2023-12-02T02:56:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 711662592 bytes 2023-12-02T02:56:52.962 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 807221248 bytes 2023-12-02T02:56:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 819166208 bytes 2023-12-02T02:56:53.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 865420288 bytes 2023-12-02T02:56:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 904812544 bytes 2023-12-02T02:56:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 917821440 bytes 2023-12-02T02:56:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 928636928 bytes 2023-12-02T02:56:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 946015232 bytes 2023-12-02T02:56:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1014530048 bytes 2023-12-02T02:56:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1019522048 bytes 2023-12-02T02:56:53.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1073576960 bytes 2023-12-02T02:56:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1140169728 bytes 2023-12-02T02:56:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1209625600 bytes 2023-12-02T02:56:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1244399616 bytes 2023-12-02T02:56:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1324155904 bytes 2023-12-02T02:56:54.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1396799488 bytes 2023-12-02T02:56:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1464885248 bytes 2023-12-02T02:56:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1527848960 bytes 2023-12-02T02:56:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1587830784 bytes 2023-12-02T02:56:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1622380544 bytes 2023-12-02T02:56:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1675970560 bytes 2023-12-02T02:56:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1736676352 bytes 2023-12-02T02:56:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1794485248 bytes 2023-12-02T02:56:55.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1830504448 bytes 2023-12-02T02:56:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1841978368 bytes 2023-12-02T02:56:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1853445120 bytes 2023-12-02T02:56:55.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 1883676672 bytes 2023-12-02T02:56:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 1955611648 bytes 2023-12-02T02:56:56.241 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2058563584 bytes 2023-12-02T02:56:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2079754240 bytes 2023-12-02T02:56:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2093553664 bytes 2023-12-02T02:56:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2136239104 bytes 2023-12-02T02:56:56.642 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2174706688 bytes 2023-12-02T02:56:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2240545792 bytes 2023-12-02T02:56:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2330394624 bytes 2023-12-02T02:56:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2375235584 bytes 2023-12-02T02:56:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2380394496 bytes 2023-12-02T02:56:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2417248256 bytes 2023-12-02T02:56:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2480902144 bytes 2023-12-02T02:56:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2521517056 bytes 2023-12-02T02:56:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2551366656 bytes 2023-12-02T02:56:57.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2605318144 bytes 2023-12-02T02:56:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2623275008 bytes 2023-12-02T02:56:57.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2666516480 bytes 2023-12-02T02:56:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2738179072 bytes 2023-12-02T02:56:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2791174144 bytes 2023-12-02T02:56:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2855603200 bytes 2023-12-02T02:56:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 2873341952 bytes 2023-12-02T02:56:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 2956053504 bytes 2023-12-02T02:56:58.720 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 2989057024 bytes 2023-12-02T02:56:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 3059565568 bytes 2023-12-02T02:56:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 3083537408 bytes 2023-12-02T02:56:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3119943680 bytes 2023-12-02T02:56:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3171446784 bytes 2023-12-02T02:56:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3213860864 bytes 2023-12-02T02:56:59.414 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3245634560 bytes 2023-12-02T02:56:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3266728960 bytes 2023-12-02T02:56:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3267347456 bytes 2023-12-02T02:56:59.799 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3402203136 bytes 2023-12-02T02:56:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3448632320 bytes 2023-12-02T02:56:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8200 IOs, 3449268224 bytes 2023-12-02T02:57:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8300 IOs, 3475021824 bytes 2023-12-02T02:57:00.208 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8400 IOs, 3507075072 bytes 2023-12-02T02:57:00.323 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8500 IOs, 3556588544 bytes 2023-12-02T02:57:00.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8600 IOs, 3590122496 bytes 2023-12-02T02:57:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8700 IOs, 3590746112 bytes 2023-12-02T02:57:00.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8800 IOs, 3620166656 bytes 2023-12-02T02:57:00.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8900 IOs, 3649551360 bytes 2023-12-02T02:57:00.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9000 IOs, 3707060224 bytes 2023-12-02T02:57:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9100 IOs, 3764680704 bytes 2023-12-02T02:57:01.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9200 IOs, 3802530816 bytes 2023-12-02T02:57:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9300 IOs, 3891374080 bytes 2023-12-02T02:57:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9400 IOs, 3951682560 bytes 2023-12-02T02:57:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T02:57:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T02:57:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T02:57:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:57:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-02T02:57:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:57:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:57:01.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:57:01.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:57:01.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:57:01.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:57:01.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:57:01.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:57:02.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T02:57:02.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:57:02.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:57:02.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:57:02.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:57:02.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:57:02.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:57:02.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T02:57:02.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T02:57:02.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:57:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:57:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:57:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T02:57:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ tee /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:57:02.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:test: 2023-12-02T02:57:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: global_id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:57:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: up+replaying 2023-12-02T02:57:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: replaying, {"bytes_per_second":17279078.4,"bytes_per_snapshot":526789120.0,"last_snapshot_bytes":518372352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701485801,"remote_snapshot_timestamp":1701485801,"replay_state":"idle"} 2023-12-02T02:57:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: service: mirror.3 on smithi057 2023-12-02T02:57:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02 02:57:01 2023-12-02T02:57:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: peer_sites: 2023-12-02T02:57:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: name: cluster2 2023-12-02T02:57:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: down+unknown 2023-12-02T02:57:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: status not found 2023-12-02T02:57:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02T02:57:02.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:57:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T02:57:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ description: .*' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:57:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":17279078.4,"bytes_per_snapshot":526789120.0,"last_snapshot_bytes":518372352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701485801,"remote_snapshot_timestamp":1701485801,"replay_state":"idle"} 2023-12-02T02:57:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T02:57:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo up+replaying 2023-12-02T02:57:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^up+' 2023-12-02T02:57:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:up+replaying 2023-12-02T02:57:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:57:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.3 on smithi057 2023-12-02T02:57:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:57:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local 'last_update=2023-12-02 02:57:01' 2023-12-02T02:57:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-02 02:57:01' 2023-12-02T02:57:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:57:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:57:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:57:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:57:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local 'prev_last_update=2023-12-02 02:57:01' 2023-12-02T02:57:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-02T02:57:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror.mirror_status 2023-12-02T02:57:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:57:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror.mirror_status 2023-12-02T02:57:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-02T02:57:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T02:57:02.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local last_update state 2023-12-02T02:57:02.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-02T02:57:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ last_update='2023-12-02 02:57:01' 2023-12-02T02:57:02.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-02T02:57:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ state=up+replaying 2023-12-02T02:57:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo up+replaying 2023-12-02T02:57:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep up+replaying 2023-12-02T02:57:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:up+replaying 2023-12-02T02:57:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:57:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:57:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap9 2023-12-02T02:57:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror test snap9 2023-12-02T02:57:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:57:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:57:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:57:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap9 2023-12-02T02:57:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2023-12-02T02:57:03.903 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8411032616182025) > weight (0.25) 2023-12-02T02:57:03.903 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7328309194607536) > weight (0.25) 2023-12-02T02:57:03.903 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9438163238616868) > weight (0.25) 2023-12-02T02:57:03.903 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-12-02T02:57:03.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-12-02T02:57:03.903 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-02T02:57:03.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi057.stderr:2023-12-02T02:57:03.902+0000 7f296178f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 91675) UID: 1000 2023-12-02T02:57:08.904 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-12-02T02:57:08.905 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T02:57:08.905 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-12-02T02:57:08.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-02T02:57:08.905 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-02T02:57:08.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-02T02:57:08.908 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2023-12-02T02:57:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-02T02:57:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:57:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:57:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:57:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-02T02:57:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:57:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:57:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:57:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T02:57:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T02:57:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:57:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:57:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-02T02:57:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:57:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:57:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:57:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T02:57:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T02:57:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T02:57:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-02T02:57:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T02:57:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T02:57:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:57:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T02:57:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T02:57:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T02:57:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:57:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T02:57:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T02:57:32.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T02:57:32.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:57:32.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T02:57:32.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T02:57:32.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:57:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:57:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T02:57:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T02:57:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:57:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:57:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:57:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T02:57:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T02:57:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:57:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:57:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T02:57:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:57:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T02:57:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T02:57:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:57:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:57:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T02:57:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T02:57:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:57:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:57:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:57:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:57:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T02:57:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:57:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:57:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T02:57:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T02:57:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T02:57:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:57:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:57:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:57:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T02:57:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:57:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:57:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:57:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:57:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-02T02:57:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T02:57:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:57:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:57:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T02:57:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T02:57:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:57:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:57:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:57:32.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T02:57:32.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T02:57:32.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:57:32.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:57:32.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T02:57:32.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:57:32.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T02:57:32.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T02:57:32.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:57:32.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:57:32.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T02:57:32.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T02:57:32.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:57:32.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:57:32.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:57:32.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:57:32.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T02:57:32.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:57:32.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:57:32.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T02:57:32.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T02:57:32.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T02:57:32.591 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:57:32.591 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:57:32.591 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:57:32.591 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-02T02:57:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:57:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:57:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:57:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:57:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-02T02:57:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T02:57:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:57:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:57:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T02:57:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T02:57:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:57:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:57:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:57:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T02:57:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T02:57:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:57:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:57:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T02:57:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:57:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T02:57:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T02:57:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:57:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:57:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T02:57:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T02:57:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:57:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:57:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:57:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T02:57:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T02:57:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:57:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:57:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T02:57:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T02:57:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T02:57:32.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:57:32.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:57:32.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T02:57:32.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-02T02:57:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:57:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:57:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:57:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:57:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-02T02:57:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T02:57:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:57:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:57:32.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T02:57:32.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T02:57:32.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:57:32.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:57:32.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:57:32.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T02:57:32.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T02:57:32.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:57:32.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:57:32.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T02:57:32.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T02:57:32.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T02:57:32.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T02:57:32.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:57:32.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:57:32.770 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T02:57:32.770 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T02:57:32.770 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:57:32.770 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:57:32.770 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:57:32.770 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T02:57:32.770 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T02:57:32.770 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:57:32.770 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:57:32.770 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T02:57:32.770 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T02:57:32.770 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T02:57:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T02:57:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T02:57:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T02:57:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-02T02:57:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:57:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:57:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:57:32.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:57:32.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 1 2023-12-02T02:57:33.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T02:57:33.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T02:57:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:57:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T02:57:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T02:57:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:57:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:57:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T02:57:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T02:57:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:57:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:57:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:57:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T02:57:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T02:57:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:57:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:57:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T02:57:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:57:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T02:57:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T02:57:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:57:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:57:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T02:57:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T02:57:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:57:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:57:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:57:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:57:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T02:57:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:57:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:57:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T02:57:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T02:57:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T02:57:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:57:33.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:57:33.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:57:33.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T02:57:33.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:57:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:57:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:57:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:57:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-02T02:57:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T02:57:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:57:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:57:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T02:57:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T02:57:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:57:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:57:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:57:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T02:57:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T02:57:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:57:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:57:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T02:57:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:57:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T02:57:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T02:57:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:57:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:57:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T02:57:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T02:57:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:57:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:57:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:57:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:57:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T02:57:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:57:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:57:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T02:57:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T02:57:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T02:57:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:57:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:57:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:57:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-02T02:57:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T02:57:34.081 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T02:57:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T02:57:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T02:57:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T02:57:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T02:57:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T02:57:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:57:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-02T02:57:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T02:57:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:57:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:57:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:57:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T02:57:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T02:57:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-02T02:57:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T02:57:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:57:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:57:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:57:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-test.status 2023-12-02T02:57:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-test.status 2023-12-02T02:57:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:57:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:57:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.status 2023-12-02T02:57:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.status 2023-12-02T02:57:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:57:34.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:57:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-02T02:57:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:57:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:57:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:57:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-02T02:57:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 37 2023-12-02T02:57:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:57:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:57:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:57:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:57:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:57:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:57:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-02T02:57:35.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T02:57:35.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=37 2023-12-02T02:57:35.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:57:35.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:57:35.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T02:57:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:57:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:57:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:57:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:57:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:57:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:57:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:57:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:57:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:57:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:57:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:57:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:57:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:57:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 37 = '' 2023-12-02T02:57:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:57:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T02:57:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:57:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:57:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:57:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:57:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:57:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:57:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:57:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:57:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:57:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:57:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:57:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:57:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:57:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 37 = '' 2023-12-02T02:57:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:57:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T02:57:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:57:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:57:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:57:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:57:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:57:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:57:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:57:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:57:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:57:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:57:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:57:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:57:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:57:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 37 = '' 2023-12-02T02:57:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:57:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1.6 2023-12-02T02:57:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:57:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:57:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:57:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:57:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:57:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:57:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:57:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:57:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=37 2023-12-02T02:57:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 37 = 37 2023-12-02T02:57:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:57:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2023-12-02T02:57:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:57:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:57:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:57:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap9 2023-12-02T02:57:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T02:57:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T02:57:38.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:57:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2023-12-02T02:57:39.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'test': 2023-12-02T02:57:39.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 512 MiB in 128 objects 2023-12-02T02:57:39.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T02:57:39.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 18 2023-12-02T02:57:39.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 10b949c95412 2023-12-02T02:57:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.10b949c95412 2023-12-02T02:57:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T02:57:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T02:57:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T02:57:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T02:57:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 02:50:03 2023 2023-12-02T02:57:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 02:56:47 2023 2023-12-02T02:57:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 02:50:03 2023 2023-12-02T02:57:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T02:57:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T02:57:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T02:57:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:57:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T02:57:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:57:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snap8-clean ']' 2023-12-02T02:57:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap8-clean 2023-12-02T02:57:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:57:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:57:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap8-clean 2023-12-02T02:57:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:57:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:57:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:57:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:57:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:57:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:57:40.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:57:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:57:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:57:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:57:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:57:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap9 2023-12-02T02:57:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:57:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:57:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap9 2023-12-02T02:57:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:57:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:57:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:57:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:57:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:57:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:57:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:57:42.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:57:42.986 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:57:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:57:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:57:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ clean_snap_name=snap9-clean 2023-12-02T02:57:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror test snap9-clean 2023-12-02T02:57:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:57:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:57:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:57:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap9-clean 2023-12-02T02:57:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2023-12-02T02:57:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:57:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror test 2023-12-02T02:57:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:57:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:57:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:57:43.483 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T02:57:43.484 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=14 2023-12-02T02:57:43.484 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T02:57:43.484 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-02T02:57:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 34589696 bytes 2023-12-02T02:57:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 55440384 bytes 2023-12-02T02:57:43.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 57597952 bytes 2023-12-02T02:57:44.075 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 141391872 bytes 2023-12-02T02:57:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 159036416 bytes 2023-12-02T02:57:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 181136384 bytes 2023-12-02T02:57:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 248932352 bytes 2023-12-02T02:57:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 331367424 bytes 2023-12-02T02:57:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 386947072 bytes 2023-12-02T02:57:44.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 468707328 bytes 2023-12-02T02:57:45.060 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 561783808 bytes 2023-12-02T02:57:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 662868992 bytes 2023-12-02T02:57:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 695585792 bytes 2023-12-02T02:57:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 749321216 bytes 2023-12-02T02:57:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 759428096 bytes 2023-12-02T02:57:45.655 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 775117824 bytes 2023-12-02T02:57:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 838038528 bytes 2023-12-02T02:57:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 874694656 bytes 2023-12-02T02:57:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 923308032 bytes 2023-12-02T02:57:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 989374464 bytes 2023-12-02T02:57:46.296 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 1021964288 bytes 2023-12-02T02:57:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 1056504832 bytes 2023-12-02T02:57:46.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 1090722816 bytes 2023-12-02T02:57:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1127995392 bytes 2023-12-02T02:57:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1151123456 bytes 2023-12-02T02:57:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1267619840 bytes 2023-12-02T02:57:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1294373888 bytes 2023-12-02T02:57:47.134 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1327164416 bytes 2023-12-02T02:57:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1394568192 bytes 2023-12-02T02:57:47.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1464137728 bytes 2023-12-02T02:57:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1514016768 bytes 2023-12-02T02:57:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1520073728 bytes 2023-12-02T02:57:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1556036608 bytes 2023-12-02T02:57:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1614741504 bytes 2023-12-02T02:57:48.007 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1669832704 bytes 2023-12-02T02:57:48.137 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1713269760 bytes 2023-12-02T02:57:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1764426752 bytes 2023-12-02T02:57:48.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1850369024 bytes 2023-12-02T02:57:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1877788672 bytes 2023-12-02T02:57:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1917771776 bytes 2023-12-02T02:57:48.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1937230848 bytes 2023-12-02T02:57:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1995932672 bytes 2023-12-02T02:57:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 2053713920 bytes 2023-12-02T02:57:49.115 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 2054334464 bytes 2023-12-02T02:57:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 2073842688 bytes 2023-12-02T02:57:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 2082681856 bytes 2023-12-02T02:57:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 2158395392 bytes 2023-12-02T02:57:49.551 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2208323584 bytes 2023-12-02T02:57:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2242591744 bytes 2023-12-02T02:57:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2304211968 bytes 2023-12-02T02:57:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2331480064 bytes 2023-12-02T02:57:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2416109568 bytes 2023-12-02T02:57:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2472022016 bytes 2023-12-02T02:57:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2523331584 bytes 2023-12-02T02:57:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2523951104 bytes 2023-12-02T02:57:50.522 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2532741120 bytes 2023-12-02T02:57:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2559807488 bytes 2023-12-02T02:57:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2614573056 bytes 2023-12-02T02:57:50.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2639504384 bytes 2023-12-02T02:57:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2664469504 bytes 2023-12-02T02:57:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2696732672 bytes 2023-12-02T02:57:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2727213056 bytes 2023-12-02T02:57:51.256 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2740003840 bytes 2023-12-02T02:57:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2750954496 bytes 2023-12-02T02:57:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2801847296 bytes 2023-12-02T02:57:51.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2848300032 bytes 2023-12-02T02:57:51.735 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2911201280 bytes 2023-12-02T02:57:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 3021654016 bytes 2023-12-02T02:57:52.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 3044882432 bytes 2023-12-02T02:57:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 3120923648 bytes 2023-12-02T02:57:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 3219281920 bytes 2023-12-02T02:57:52.461 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 3233239040 bytes 2023-12-02T02:57:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 3272052736 bytes 2023-12-02T02:57:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3334428672 bytes 2023-12-02T02:57:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3408772096 bytes 2023-12-02T02:57:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3527014400 bytes 2023-12-02T02:57:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3576940544 bytes 2023-12-02T02:57:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3646038016 bytes 2023-12-02T02:57:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3664663552 bytes 2023-12-02T02:57:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3715871744 bytes 2023-12-02T02:57:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3740584960 bytes 2023-12-02T02:57:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8200 IOs, 3747295232 bytes 2023-12-02T02:57:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8300 IOs, 3804772352 bytes 2023-12-02T02:57:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8400 IOs, 3805406208 bytes 2023-12-02T02:57:54.096 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8500 IOs, 3840872448 bytes 2023-12-02T02:57:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8600 IOs, 3901209600 bytes 2023-12-02T02:57:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8700 IOs, 3911390208 bytes 2023-12-02T02:57:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8800 IOs, 3932968960 bytes 2023-12-02T02:57:54.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8900 IOs, 4033920000 bytes 2023-12-02T02:57:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9000 IOs, 4088324096 bytes 2023-12-02T02:57:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9100 IOs, 4151698432 bytes 2023-12-02T02:57:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9200 IOs, 4186416128 bytes 2023-12-02T02:57:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9300 IOs, 4242310144 bytes 2023-12-02T02:57:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9400 IOs, 4275294208 bytes 2023-12-02T02:57:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9500 IOs, 4291734528 bytes 2023-12-02T02:57:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9600 IOs, 4331236352 bytes 2023-12-02T02:57:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9700 IOs, 4331852800 bytes 2023-12-02T02:57:55.692 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9800 IOs, 4407096320 bytes 2023-12-02T02:57:55.809 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9900 IOs, 4445556736 bytes 2023-12-02T02:57:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10000 IOs, 4480700416 bytes 2023-12-02T02:57:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10100 IOs, 4538743808 bytes 2023-12-02T02:57:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10200 IOs, 4553647104 bytes 2023-12-02T02:57:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10300 IOs, 4589108224 bytes 2023-12-02T02:57:56.340 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10400 IOs, 4637818880 bytes 2023-12-02T02:57:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10500 IOs, 4638440448 bytes 2023-12-02T02:57:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10600 IOs, 4688581632 bytes 2023-12-02T02:57:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10700 IOs, 4723205120 bytes 2023-12-02T02:57:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10800 IOs, 4779226112 bytes 2023-12-02T02:57:56.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10900 IOs, 4806405120 bytes 2023-12-02T02:57:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11000 IOs, 4900297728 bytes 2023-12-02T02:57:57.226 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11100 IOs, 4944164864 bytes 2023-12-02T02:57:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11200 IOs, 5007357952 bytes 2023-12-02T02:57:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T02:57:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T02:57:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T02:57:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:57:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-02T02:57:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:57:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:57:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:57:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:57:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:57:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:57:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:57:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:57:58.490 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-02T02:57:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:57:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:57:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:57:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:57:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local description_pattern= 2023-12-02T02:57:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local service_pattern= 2023-12-02T02:57:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-02T02:57:58.492 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-02T02:57:58.492 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:57:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:57:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T02:57:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-02T02:57:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ tee /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:57:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:test: 2023-12-02T02:57:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: global_id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:57:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: up+replaying 2023-12-02T02:57:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701485801,"remote_snapshot_timestamp":1701485801,"replay_state":"idle"} 2023-12-02T02:57:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: service: mirror.1 on smithi057 2023-12-02T02:57:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02 02:57:33 2023-12-02T02:57:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: peer_sites: 2023-12-02T02:57:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: name: cluster2 2023-12-02T02:57:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: state: up+stopped 2023-12-02T02:57:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: description: local image is primary 2023-12-02T02:57:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: last_update: 2023-12-02 02:57:46 2023-12-02T02:57:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:57:58.591 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T02:57:58.591 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^ description: .*' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:57:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701485801,"remote_snapshot_timestamp":1701485801,"replay_state":"idle"} 2023-12-02T02:57:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T02:57:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo up+replaying 2023-12-02T02:57:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep '^up+' 2023-12-02T02:57:58.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:up+replaying 2023-12-02T02:57:58.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:57:58.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.1 on smithi057 2023-12-02T02:57:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.kebs98RNzE/cluster1-mirror-test.mirror_status 2023-12-02T02:57:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local 'last_update=2023-12-02 02:57:33' 2023-12-02T02:57:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-02 02:57:33' 2023-12-02T02:57:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:57:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:57:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:57:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state_pattern=up+replaying 2023-12-02T02:57:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local 'prev_last_update=2023-12-02 02:57:33' 2023-12-02T02:57:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-02T02:57:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror.mirror_status 2023-12-02T02:57:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:57:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror.mirror_status 2023-12-02T02:57:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-02T02:57:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T02:57:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local last_update state 2023-12-02T02:57:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-02T02:57:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ last_update='2023-12-02 02:57:33' 2023-12-02T02:57:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-02T02:57:58.749 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ state=up+replaying 2023-12-02T02:57:58.749 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo up+replaying 2023-12-02T02:57:58.749 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep up+replaying 2023-12-02T02:57:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:up+replaying 2023-12-02T02:57:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:57:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:57:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap10 2023-12-02T02:57:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror test snap10 2023-12-02T02:57:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:57:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:57:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:57:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap10 2023-12-02T02:57:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2023-12-02T02:58:09.007 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-02T02:58:09.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-02T02:58:09.008 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-02T02:58:09.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr:2023-12-02T02:58:09.006+0000 7f35cd8b1640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 97609) UID: 1000 2023-12-02T02:58:10.009 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-02T02:58:10.009 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T02:58:10.009 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-02T02:58:10.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-02T02:58:10.010 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-02T02:58:10.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-02T02:58:10.013 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2023-12-02T02:58:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-02T02:58:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:58:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-02T02:58:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:58:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T02:58:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T02:58:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:58:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:58:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-02T02:58:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:58:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T02:58:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T02:58:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T02:58:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-02T02:58:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T02:58:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:58:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T02:58:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T02:58:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T02:58:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T02:58:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:58:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T02:58:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T02:58:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T02:58:27.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:58:27.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T02:58:27.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T02:58:27.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:58:27.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:58:27.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T02:58:27.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T02:58:27.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:58:27.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:58:27.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:58:27.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T02:58:27.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T02:58:27.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:58:27.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:58:27.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T02:58:27.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:58:27.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T02:58:27.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T02:58:27.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:58:27.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:58:27.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T02:58:27.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T02:58:27.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:58:27.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:58:27.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:58:27.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:58:27.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T02:58:27.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:58:27.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:58:27.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T02:58:27.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T02:58:27.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T02:58:27.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:58:27.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:58:27.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:58:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T02:58:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:58:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:58:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:58:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:58:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-02T02:58:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T02:58:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:58:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:58:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T02:58:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T02:58:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:58:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:58:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:58:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T02:58:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T02:58:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:58:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:58:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T02:58:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:58:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T02:58:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T02:58:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:58:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:58:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T02:58:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T02:58:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:58:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:58:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:58:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:58:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T02:58:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:58:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:58:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T02:58:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T02:58:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T02:58:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:58:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:58:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:58:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-02T02:58:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T02:58:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T02:58:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T02:58:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T02:58:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T02:58:27.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T02:58:27.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T02:58:27.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:58:27.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-02T02:58:27.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T02:58:27.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:58:27.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:27.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:27.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T02:58:27.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T02:58:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-02T02:58:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T02:58:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:58:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-test.status 2023-12-02T02:58:27.729 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-test.status 2023-12-02T02:58:27.729 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:58:27.730 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:58:27.730 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.status 2023-12-02T02:58:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.status 2023-12-02T02:58:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:58:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:58:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-02T02:58:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:58:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-02T02:58:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 41 2023-12-02T02:58:29.549 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:58:29.550 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:58:29.550 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:29.550 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:29.550 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:58:29.550 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:58:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-02T02:58:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T02:58:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=41 2023-12-02T02:58:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:58:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:58:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T02:58:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:58:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:58:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:58:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:58:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:58:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:58:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:58:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:58:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:58:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:58:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:58:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 41 = '' 2023-12-02T02:58:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:58:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T02:58:30.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:58:30.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:58:30.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:30.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:30.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:58:30.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:58:30.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:58:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:58:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:58:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:58:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:58:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:58:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:58:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 41 = '' 2023-12-02T02:58:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:58:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T02:58:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:58:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:58:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:58:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:58:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:58:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:58:31.281 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:58:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:58:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:58:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:58:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:58:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 41 = '' 2023-12-02T02:58:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:58:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1.6 2023-12-02T02:58:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:58:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:58:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:58:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:58:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:58:32.909 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7752929105471245) > weight (0.25) 2023-12-02T02:58:32.909 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7440882027785464) > weight (0.25) 2023-12-02T02:58:32.909 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.3706481665411048) > weight (0.25) 2023-12-02T02:58:32.910 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6797456832975792) > weight (0.25) 2023-12-02T02:58:32.910 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2023-12-02T02:58:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:58:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=41 2023-12-02T02:58:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 41 = 41 2023-12-02T02:58:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:58:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2023-12-02T02:58:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:58:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap10 2023-12-02T02:58:32.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T02:58:32.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T02:58:32.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:58:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2023-12-02T02:58:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'test': 2023-12-02T02:58:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 512 MiB in 128 objects 2023-12-02T02:58:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T02:58:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 20 2023-12-02T02:58:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 10b949c95412 2023-12-02T02:58:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.10b949c95412 2023-12-02T02:58:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T02:58:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T02:58:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T02:58:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T02:58:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 02:50:03 2023 2023-12-02T02:58:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 02:56:47 2023 2023-12-02T02:58:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 02:50:03 2023 2023-12-02T02:58:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T02:58:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T02:58:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T02:58:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:58:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T02:58:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:58:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snap9-clean ']' 2023-12-02T02:58:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap9-clean 2023-12-02T02:58:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap9-clean 2023-12-02T02:58:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:58:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:58:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:58:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:58:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:58:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:58:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap10 2023-12-02T02:58:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap10 2023-12-02T02:58:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:58:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:58:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:58:36.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:58:36.276 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:58:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:37.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:58:37.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ clean_snap_name=snap10-clean 2023-12-02T02:58:37.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror test snap10-clean 2023-12-02T02:58:37.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:58:37.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:37.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:37.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap10-clean 2023-12-02T02:58:37.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2023-12-02T02:58:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-02T02:58:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:58:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-02T02:58:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:58:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T02:58:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T02:58:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:58:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:58:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-02T02:58:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:58:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T02:58:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T02:58:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T02:58:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-02T02:58:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T02:58:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T02:58:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:58:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T02:58:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T02:58:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T02:58:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T02:58:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T02:58:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T02:58:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T02:58:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:58:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-02T02:58:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T02:58:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:58:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:58:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T02:58:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T02:58:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:58:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:58:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:58:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T02:58:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T02:58:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:58:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:58:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T02:58:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:58:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T02:58:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T02:58:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:58:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:58:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T02:58:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T02:58:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:58:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:58:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:58:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T02:58:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T02:58:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:58:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:58:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T02:58:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T02:58:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T02:58:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:58:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:58:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T02:58:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-02T02:58:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-02T02:58:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T02:58:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T02:58:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T02:58:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-02T02:58:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T02:58:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T02:58:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T02:58:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T02:58:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T02:58:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T02:58:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T02:58:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T02:58:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T02:58:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T02:58:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T02:58:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:58:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T02:58:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T02:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T02:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T02:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T02:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T02:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T02:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T02:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T02:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T02:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T02:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T02:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T02:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T02:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T02:58:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T02:58:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T02:58:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:58:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:58:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T02:58:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-02T02:58:39.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T02:58:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T02:58:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T02:58:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T02:58:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T02:58:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T02:58:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T02:58:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:58:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-02T02:58:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T02:58:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:58:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T02:58:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T02:58:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-02T02:58:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T02:58:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:58:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-test.status 2023-12-02T02:58:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-test.status 2023-12-02T02:58:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:58:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:58:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-test.status 2023-12-02T02:58:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-test.status 2023-12-02T02:58:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T02:58:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:58:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-02T02:58:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:58:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-02T02:58:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 44 2023-12-02T02:58:41.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:58:41.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:58:41.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:41.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:41.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.status 2023-12-02T02:58:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:58:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-02T02:58:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T02:58:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=44 2023-12-02T02:58:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:58:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:58:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T02:58:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:58:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:58:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:58:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:58:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:58:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:58:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:58:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:58:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:58:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:58:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:58:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 44 = '' 2023-12-02T02:58:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:58:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T02:58:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:58:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:58:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:58:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:58:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:58:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T02:58:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:58:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T02:58:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T02:58:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T02:58:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T02:58:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 44 = '' 2023-12-02T02:58:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T02:58:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T02:58:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:58:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:58:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.status 2023-12-02T02:58:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-02T02:58:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T02:58:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T02:58:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=44 2023-12-02T02:58:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 44 = 44 2023-12-02T02:58:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:58:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2023-12-02T02:58:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:58:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap10-clean 2023-12-02T02:58:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T02:58:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T02:58:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T02:58:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2023-12-02T02:58:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'test': 2023-12-02T02:58:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 512 MiB in 128 objects 2023-12-02T02:58:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T02:58:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 21 2023-12-02T02:58:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 10b949c95412 2023-12-02T02:58:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.10b949c95412 2023-12-02T02:58:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T02:58:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T02:58:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T02:58:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T02:58:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 02:50:03 2023 2023-12-02T02:58:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 02:58:34 2023 2023-12-02T02:58:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 02:50:03 2023 2023-12-02T02:58:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T02:58:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T02:58:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T02:58:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: b6d7f94d-a85d-4f49-9985-9d8fb2898594 2023-12-02T02:58:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T02:58:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:58:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ seq 1 10 2023-12-02T02:58:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:58:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap1 2023-12-02T02:58:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap1 2023-12-02T02:58:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap1 2023-12-02T02:58:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:58:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:58:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:58:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:58:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:45.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:58:45.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:58:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap1-clean 2023-12-02T02:58:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap1-clean 2023-12-02T02:58:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap1-clean 2023-12-02T02:58:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:58:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:58:45.872 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:45.872 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:45.872 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:58:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:58:46.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:47.599 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:58:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:47.858 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:58:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:58:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap2 2023-12-02T02:58:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap2 2023-12-02T02:58:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap2 2023-12-02T02:58:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:58:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:58:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:58:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:58:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:58:49.319 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:58:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap2-clean 2023-12-02T02:58:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap2-clean 2023-12-02T02:58:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap2-clean 2023-12-02T02:58:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:58:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:58:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:58:50.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:58:50.276 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:58:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:51.288 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:58:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:58:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap3 2023-12-02T02:58:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap3 2023-12-02T02:58:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap3 2023-12-02T02:58:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:58:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:58:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:58:51.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:58:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:58:52.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:53.015 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:58:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap3-clean 2023-12-02T02:58:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap3-clean 2023-12-02T02:58:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:53.138 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap3-clean 2023-12-02T02:58:53.138 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:58:53.138 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:58:53.138 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:53.138 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:53.138 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:58:53.699 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:58:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:58:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:54.797 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:58:54.797 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:58:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap4 2023-12-02T02:58:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap4 2023-12-02T02:58:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap4 2023-12-02T02:58:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:58:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:58:54.799 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:54.799 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:54.799 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:58:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:58:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:58:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:56.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:58:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap4-clean 2023-12-02T02:58:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap4-clean 2023-12-02T02:58:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap4-clean 2023-12-02T02:58:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:58:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:58:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:58:57.072 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:58:57.083 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:58:57.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:58:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:58:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap5 2023-12-02T02:58:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap5 2023-12-02T02:58:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap5 2023-12-02T02:58:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:58:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:58:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:58:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:58:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:58:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:58:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap5-clean 2023-12-02T02:58:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap5-clean 2023-12-02T02:58:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:58:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:58:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap5-clean 2023-12-02T02:58:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:58:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:58:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:58:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:59:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:59:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:01.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:59:01.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:01.697 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:59:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:59:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap6 2023-12-02T02:59:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap6 2023-12-02T02:59:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap6 2023-12-02T02:59:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:59:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:59:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:59:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:59:02.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:59:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:03.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:59:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap6-clean 2023-12-02T02:59:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap6-clean 2023-12-02T02:59:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap6-clean 2023-12-02T02:59:03.531 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:59:03.531 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:59:03.531 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:03.531 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:03.531 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:59:04.101 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:59:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:59:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:05.146 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:59:05.146 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:59:05.146 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap7 2023-12-02T02:59:05.146 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap7 2023-12-02T02:59:05.146 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:05.146 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap7 2023-12-02T02:59:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:59:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:59:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:59:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:59:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:59:06.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:59:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap7-clean 2023-12-02T02:59:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap7-clean 2023-12-02T02:59:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap7-clean 2023-12-02T02:59:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:59:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:59:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:59:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:59:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:08.061 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:59:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:59:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:59:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap8 2023-12-02T02:59:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap8 2023-12-02T02:59:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap8 2023-12-02T02:59:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:59:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:59:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:59:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:59:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:59:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:10.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:59:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap8-clean 2023-12-02T02:59:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap8-clean 2023-12-02T02:59:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap8-clean 2023-12-02T02:59:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:59:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:59:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:59:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:59:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:11.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:59:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:11.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:59:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:59:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap9 2023-12-02T02:59:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap9 2023-12-02T02:59:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap9 2023-12-02T02:59:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:59:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:59:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:59:12.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:59:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:59:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:59:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap9-clean 2023-12-02T02:59:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap9-clean 2023-12-02T02:59:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap9-clean 2023-12-02T02:59:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:59:13.464 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:59:13.464 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:13.464 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:13.464 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:59:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:59:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:14.852 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:59:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:59:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:59:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap10 2023-12-02T02:59:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap10 2023-12-02T02:59:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap10 2023-12-02T02:59:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:59:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:59:15.230 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:15.230 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:15.230 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:59:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:59:15.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:16.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:59:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:16.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:59:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap10-clean 2023-12-02T02:59:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror test snap10-clean 2023-12-02T02:59:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap10-clean 2023-12-02T02:59:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T02:59:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:59:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export 2023-12-02T02:59:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:59:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-02T02:59:18.207 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-test.export /tmp/tmp.kebs98RNzE/cluster1-mirror-test.export 2023-12-02T02:59:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:59:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ seq 1 10 2023-12-02T02:59:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:59:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap1 2023-12-02T02:59:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ remove_snapshot cluster2 mirror test snap1 2023-12-02T02:59:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap=snap1 2023-12-02T02:59:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2023-12-02T02:59:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Removing snap: 100% complete...done. 2023-12-02T02:59:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap1-clean 2023-12-02T02:59:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2023-12-02T02:59:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap=snap1-clean 2023-12-02T02:59:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2023-12-02T02:59:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Removing snap: 100% complete...done. 2023-12-02T02:59:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:59:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap2 2023-12-02T02:59:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ remove_snapshot cluster2 mirror test snap2 2023-12-02T02:59:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap=snap2 2023-12-02T02:59:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2023-12-02T02:59:21.484 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Removing snap: 100% complete...done. 2023-12-02T02:59:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap2-clean 2023-12-02T02:59:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2023-12-02T02:59:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:21.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:21.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap=snap2-clean 2023-12-02T02:59:21.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2023-12-02T02:59:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Removing snap: 100% complete...done. 2023-12-02T02:59:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:59:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap3 2023-12-02T02:59:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ remove_snapshot cluster2 mirror test snap3 2023-12-02T02:59:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap=snap3 2023-12-02T02:59:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2023-12-02T02:59:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Removing snap: 100% complete...done. 2023-12-02T02:59:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap3-clean 2023-12-02T02:59:23.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2023-12-02T02:59:23.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:23.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:23.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:23.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap=snap3-clean 2023-12-02T02:59:23.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2023-12-02T02:59:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Removing snap: 100% complete...done. 2023-12-02T02:59:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:59:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap4 2023-12-02T02:59:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ remove_snapshot cluster2 mirror test snap4 2023-12-02T02:59:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap=snap4 2023-12-02T02:59:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2023-12-02T02:59:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Removing snap: 100% complete...done. 2023-12-02T02:59:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap4-clean 2023-12-02T02:59:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2023-12-02T02:59:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap=snap4-clean 2023-12-02T02:59:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2023-12-02T02:59:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Removing snap: 100% complete...done. 2023-12-02T02:59:26.551 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:59:26.551 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap5 2023-12-02T02:59:26.551 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ remove_snapshot cluster2 mirror test snap5 2023-12-02T02:59:26.551 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:26.551 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:26.551 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:26.551 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap=snap5 2023-12-02T02:59:26.551 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2023-12-02T02:59:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Removing snap: 100% complete...done. 2023-12-02T02:59:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap5-clean 2023-12-02T02:59:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2023-12-02T02:59:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap=snap5-clean 2023-12-02T02:59:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2023-12-02T02:59:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Removing snap: 100% complete...done. 2023-12-02T02:59:28.562 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:59:28.563 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap6 2023-12-02T02:59:28.563 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ remove_snapshot cluster2 mirror test snap6 2023-12-02T02:59:28.563 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:28.563 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:28.563 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:28.563 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap=snap6 2023-12-02T02:59:28.563 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2023-12-02T02:59:29.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Removing snap: 100% complete...done. 2023-12-02T02:59:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap6-clean 2023-12-02T02:59:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2023-12-02T02:59:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap=snap6-clean 2023-12-02T02:59:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2023-12-02T02:59:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Removing snap: 100% complete...done. 2023-12-02T02:59:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:59:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap7 2023-12-02T02:59:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ remove_snapshot cluster2 mirror test snap7 2023-12-02T02:59:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap=snap7 2023-12-02T02:59:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2023-12-02T02:59:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Removing snap: 100% complete...done. 2023-12-02T02:59:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap7-clean 2023-12-02T02:59:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2023-12-02T02:59:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap=snap7-clean 2023-12-02T02:59:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2023-12-02T02:59:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Removing snap: 100% complete...done. 2023-12-02T02:59:32.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:59:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap8 2023-12-02T02:59:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ remove_snapshot cluster2 mirror test snap8 2023-12-02T02:59:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap=snap8 2023-12-02T02:59:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2023-12-02T02:59:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Removing snap: 100% complete...done. 2023-12-02T02:59:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap8-clean 2023-12-02T02:59:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2023-12-02T02:59:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap=snap8-clean 2023-12-02T02:59:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2023-12-02T02:59:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Removing snap: 100% complete...done. 2023-12-02T02:59:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:59:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap9 2023-12-02T02:59:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ remove_snapshot cluster2 mirror test snap9 2023-12-02T02:59:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap=snap9 2023-12-02T02:59:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2023-12-02T02:59:35.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Removing snap: 100% complete...done. 2023-12-02T02:59:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap9-clean 2023-12-02T02:59:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2023-12-02T02:59:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap=snap9-clean 2023-12-02T02:59:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2023-12-02T02:59:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Removing snap: 100% complete...done. 2023-12-02T02:59:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 10` 2023-12-02T02:59:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap10 2023-12-02T02:59:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ remove_snapshot cluster2 mirror test snap10 2023-12-02T02:59:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap=snap10 2023-12-02T02:59:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2023-12-02T02:59:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Removing snap: 100% complete...done. 2023-12-02T02:59:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap10-clean 2023-12-02T02:59:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2023-12-02T02:59:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap=snap10-clean 2023-12-02T02:59:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2023-12-02T02:59:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Removing snap: 100% complete...done. 2023-12-02T02:59:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ remove_image_retry cluster2 mirror test 2023-12-02T02:59:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-02T02:59:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0 2023-12-02T02:59:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ remove_image cluster2 mirror test 2023-12-02T02:59:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2023-12-02T02:59:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 rm mirror/test 2023-12-02T02:59:38.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi057.stderr:2023-12-02T02:59:38.745+0000 7f3425ceb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d8b10d41a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-02T02:59:38.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi057.stderr:2023-12-02T02:59:38.745+0000 7f342ccf9640 -1 rbd::mirror::ImageReplayer: 0x55d8b030e280 [3/b6d7f94d-a85d-4f49-9985-9d8fb2898594] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-02T02:59:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-02T02:59:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:59:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_present cluster1 mirror test deleted 2023-12-02T02:59:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:59:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=deleted 2023-12-02T02:59:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image_id= 2023-12-02T02:59:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T02:59:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test -n '' 2023-12-02T02:59:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ get_image_id cluster1 mirror test 2023-12-02T02:59:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T02:59:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local pool=mirror 2023-12-02T02:59:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local image=test 2023-12-02T02:59:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ rbd --cluster=cluster1 info mirror/test 2023-12-02T02:59:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T02:59:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: error opening image test: (2) No such file or directory 2023-12-02T02:59:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image_id= 2023-12-02T02:59:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-02T02:59:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.1 2023-12-02T02:59:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_present cluster1 mirror test deleted '' 2023-12-02T02:59:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T02:59:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=test 2023-12-02T02:59:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=deleted 2023-12-02T02:59:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image_id= 2023-12-02T02:59:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=deleted 2023-12-02T02:59:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_image_id 2023-12-02T02:59:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ get_image_id cluster1 mirror test 2023-12-02T02:59:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T02:59:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local pool=mirror 2023-12-02T02:59:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local image=test 2023-12-02T02:59:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ rbd --cluster=cluster1 info mirror/test 2023-12-02T02:59:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-02T02:59:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: error opening image test: (2) No such file or directory 2023-12-02T02:59:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_image_id= 2023-12-02T02:59:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test -n '' 2023-12-02T02:59:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test deleted = deleted 2023-12-02T02:59:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:59:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ testlog 'TEST: create many images' 2023-12-02T02:59:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ tee -a /tmp/tmp.kebs98RNzE/rbd-mirror.test.log 2023-12-02T02:59:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ date '+%F %T' 2023-12-02T02:59:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02 02:59:39 TEST: create many images 2023-12-02T02:59:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:2023-12-02 02:59:39 TEST: create many images 2023-12-02T02:59:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ snap_name=snap 2023-12-02T02:59:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ seq 1 50 2023-12-02T02:59:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T02:59:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_1 2023-12-02T02:59:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2023-12-02T02:59:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:59:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:59:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_1 2023-12-02T02:59:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:59:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T02:59:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T02:59:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:59:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_1 128M 2023-12-02T02:59:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:59:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:59:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_1 2023-12-02T02:59:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:59:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T02:59:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T02:59:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T02:59:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:59:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_1 2023-12-02T02:59:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T02:59:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T02:59:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:59:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2023-12-02T02:59:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_1 2023-12-02T02:59:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T02:59:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2023-12-02T02:59:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T02:59:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2023-12-02T02:59:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_1': 2023-12-02T02:59:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T02:59:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T02:59:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T02:59:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 127da273cdda 2023-12-02T02:59:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.127da273cdda 2023-12-02T02:59:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T02:59:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T02:59:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T02:59:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T02:59:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 02:59:39 2023 2023-12-02T02:59:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 02:59:39 2023 2023-12-02T02:59:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 02:59:39 2023 2023-12-02T02:59:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T02:59:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T02:59:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 49c16484-3472-4671-8be8-03c1846564a3 2023-12-02T02:59:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T02:59:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T02:59:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_1 2023-12-02T02:59:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_1 2023-12-02T02:59:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T02:59:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=11 2023-12-02T02:59:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T02:59:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2023-12-02T02:59:40.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 31632384 bytes 2023-12-02T02:59:40.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 84456448 bytes 2023-12-02T02:59:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 139010048 bytes 2023-12-02T02:59:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 161232896 bytes 2023-12-02T02:59:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 180362240 bytes 2023-12-02T02:59:41.307 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 219066368 bytes 2023-12-02T02:59:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 266957824 bytes 2023-12-02T02:59:41.522 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 293740544 bytes 2023-12-02T02:59:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 331790336 bytes 2023-12-02T02:59:41.757 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 386405376 bytes 2023-12-02T02:59:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 433133568 bytes 2023-12-02T02:59:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 463419392 bytes 2023-12-02T02:59:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 489378816 bytes 2023-12-02T02:59:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 511562752 bytes 2023-12-02T02:59:42.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 524856320 bytes 2023-12-02T02:59:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 627066880 bytes 2023-12-02T02:59:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 643910656 bytes 2023-12-02T02:59:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 652909568 bytes 2023-12-02T02:59:42.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 699260928 bytes 2023-12-02T02:59:42.796 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 699880448 bytes 2023-12-02T02:59:42.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 747387904 bytes 2023-12-02T02:59:42.985 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 747981824 bytes 2023-12-02T02:59:43.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 776152064 bytes 2023-12-02T02:59:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 795684864 bytes 2023-12-02T02:59:43.255 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 813954048 bytes 2023-12-02T02:59:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 838653952 bytes 2023-12-02T02:59:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 858740736 bytes 2023-12-02T02:59:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 870945792 bytes 2023-12-02T02:59:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 896308224 bytes 2023-12-02T02:59:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 918651904 bytes 2023-12-02T02:59:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 937229312 bytes 2023-12-02T02:59:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 987042816 bytes 2023-12-02T02:59:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1068848128 bytes 2023-12-02T02:59:44.244 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1118711808 bytes 2023-12-02T02:59:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1188146176 bytes 2023-12-02T02:59:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1223523328 bytes 2023-12-02T02:59:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1284194304 bytes 2023-12-02T02:59:44.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1337684992 bytes 2023-12-02T02:59:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1408391168 bytes 2023-12-02T02:59:45.014 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5188927561544713) > weight (0.25) 2023-12-02T02:59:45.014 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9130951630495469) > weight (0.25) 2023-12-02T02:59:45.014 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-02T02:59:45.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-02T02:59:45.015 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-02T02:59:45.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi057.stderr:2023-12-02T02:59:45.014+0000 7f342d4fa640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 93634) UID: 1000 2023-12-02T02:59:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1488827392 bytes 2023-12-02T02:59:45.233 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1531624448 bytes 2023-12-02T02:59:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1541692416 bytes 2023-12-02T02:59:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1553061888 bytes 2023-12-02T02:59:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1562690560 bytes 2023-12-02T02:59:45.549 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1573049344 bytes 2023-12-02T02:59:45.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1582605312 bytes 2023-12-02T02:59:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1599761408 bytes 2023-12-02T02:59:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 1628010496 bytes 2023-12-02T02:59:45.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 1654771712 bytes 2023-12-02T02:59:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 1695771648 bytes 2023-12-02T02:59:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 1773698048 bytes 2023-12-02T02:59:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 1824544768 bytes 2023-12-02T02:59:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 1862547456 bytes 2023-12-02T02:59:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 1874572288 bytes 2023-12-02T02:59:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 1895634944 bytes 2023-12-02T02:59:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 1928406016 bytes 2023-12-02T02:59:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2012120064 bytes 2023-12-02T02:59:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2103112704 bytes 2023-12-02T02:59:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2133520384 bytes 2023-12-02T02:59:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2193752064 bytes 2023-12-02T02:59:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2257789952 bytes 2023-12-02T02:59:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2289718272 bytes 2023-12-02T02:59:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2331906048 bytes 2023-12-02T02:59:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2363908096 bytes 2023-12-02T02:59:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2411905024 bytes 2023-12-02T02:59:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2452345856 bytes 2023-12-02T02:59:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2468147200 bytes 2023-12-02T02:59:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2495366144 bytes 2023-12-02T02:59:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 2561012736 bytes 2023-12-02T02:59:48.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 2608685056 bytes 2023-12-02T02:59:48.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 2609288192 bytes 2023-12-02T02:59:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 2653395968 bytes 2023-12-02T02:59:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 2716291072 bytes 2023-12-02T02:59:49.015 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-02T02:59:49.016 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T02:59:49.016 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-02T02:59:49.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-02T02:59:49.017 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-02T02:59:49.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-02T02:59:49.025 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2023-12-02T02:59:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 2801156096 bytes 2023-12-02T02:59:49.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 2908378112 bytes 2023-12-02T02:59:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 2943970304 bytes 2023-12-02T02:59:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 2977665024 bytes 2023-12-02T02:59:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3037181952 bytes 2023-12-02T02:59:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3061904384 bytes 2023-12-02T02:59:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3108143104 bytes 2023-12-02T02:59:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3111795712 bytes 2023-12-02T02:59:50.031 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8200 IOs, 3150024704 bytes 2023-12-02T02:59:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8300 IOs, 3198899200 bytes 2023-12-02T02:59:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8400 IOs, 3266562048 bytes 2023-12-02T02:59:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8500 IOs, 3298126848 bytes 2023-12-02T02:59:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8600 IOs, 3339471872 bytes 2023-12-02T02:59:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8700 IOs, 3390922752 bytes 2023-12-02T02:59:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8800 IOs, 3414223872 bytes 2023-12-02T02:59:50.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8900 IOs, 3456649216 bytes 2023-12-02T02:59:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9000 IOs, 3467110400 bytes 2023-12-02T02:59:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9100 IOs, 3467748352 bytes 2023-12-02T02:59:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9200 IOs, 3537180672 bytes 2023-12-02T02:59:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9300 IOs, 3583613952 bytes 2023-12-02T02:59:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9400 IOs, 3657408512 bytes 2023-12-02T02:59:51.549 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9500 IOs, 3671409664 bytes 2023-12-02T02:59:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T02:59:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T02:59:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T02:59:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:59:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T02:59:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_2 2023-12-02T02:59:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2023-12-02T02:59:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:59:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:59:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_2 2023-12-02T02:59:51.660 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:59:51.660 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T02:59:51.660 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T02:59:51.660 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:59:51.660 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_2 128M 2023-12-02T02:59:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:59:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:59:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_2 2023-12-02T02:59:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:59:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T02:59:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T02:59:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T02:59:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T02:59:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_2 2023-12-02T02:59:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T02:59:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T02:59:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T02:59:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2023-12-02T02:59:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_2 2023-12-02T02:59:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T02:59:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2023-12-02T02:59:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T02:59:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2023-12-02T02:59:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_2': 2023-12-02T02:59:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T02:59:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T02:59:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T02:59:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 1291d37e6866 2023-12-02T02:59:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.1291d37e6866 2023-12-02T02:59:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T02:59:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T02:59:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T02:59:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T02:59:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 02:59:51 2023 2023-12-02T02:59:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 02:59:51 2023 2023-12-02T02:59:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 02:59:51 2023 2023-12-02T02:59:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T02:59:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T02:59:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 09e06fe9-a175-4009-952b-403e63e1ac10 2023-12-02T02:59:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T02:59:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T02:59:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_2 2023-12-02T02:59:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T02:59:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T02:59:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_2 2023-12-02T02:59:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T02:59:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=11 2023-12-02T02:59:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T02:59:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2023-12-02T02:59:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 31932416 bytes 2023-12-02T02:59:52.770 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 73062400 bytes 2023-12-02T02:59:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 148586496 bytes 2023-12-02T02:59:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 196042752 bytes 2023-12-02T02:59:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 256870400 bytes 2023-12-02T02:59:53.327 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 328492032 bytes 2023-12-02T02:59:53.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 374756352 bytes 2023-12-02T02:59:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 407126016 bytes 2023-12-02T02:59:53.657 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 439087104 bytes 2023-12-02T02:59:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 488256512 bytes 2023-12-02T02:59:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 553859072 bytes 2023-12-02T02:59:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 586045440 bytes 2023-12-02T02:59:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 624323584 bytes 2023-12-02T02:59:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 669296640 bytes 2023-12-02T02:59:54.380 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 733395968 bytes 2023-12-02T02:59:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 800661504 bytes 2023-12-02T02:59:54.642 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 836068352 bytes 2023-12-02T02:59:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 863206400 bytes 2023-12-02T02:59:54.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 926277632 bytes 2023-12-02T02:59:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 926876672 bytes 2023-12-02T02:59:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 956598272 bytes 2023-12-02T02:59:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 972916736 bytes 2023-12-02T02:59:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 994656256 bytes 2023-12-02T02:59:55.366 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1069556736 bytes 2023-12-02T02:59:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1095451648 bytes 2023-12-02T02:59:55.639 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1157652480 bytes 2023-12-02T02:59:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1179033600 bytes 2023-12-02T02:59:55.854 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1228925952 bytes 2023-12-02T02:59:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1283752960 bytes 2023-12-02T02:59:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1334449152 bytes 2023-12-02T02:59:56.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1353094144 bytes 2023-12-02T02:59:56.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1375306752 bytes 2023-12-02T02:59:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1409636352 bytes 2023-12-02T02:59:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1436276736 bytes 2023-12-02T02:59:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1487937536 bytes 2023-12-02T02:59:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1538013184 bytes 2023-12-02T02:59:56.911 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-02T02:59:56.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-02T02:59:56.911 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-02T02:59:56.911 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-02T02:59:56.911 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T02:59:56.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi057.stderr:2023-12-02T02:59:56.910+0000 7ff2cca39640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 95239) UID: 1000 2023-12-02T02:59:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1578645504 bytes 2023-12-02T02:59:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1612214272 bytes 2023-12-02T02:59:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1632239616 bytes 2023-12-02T02:59:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1648234496 bytes 2023-12-02T02:59:57.366 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1688690688 bytes 2023-12-02T02:59:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1705396224 bytes 2023-12-02T02:59:57.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1732623360 bytes 2023-12-02T02:59:57.656 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1753711616 bytes 2023-12-02T02:59:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1754329088 bytes 2023-12-02T02:59:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1852213248 bytes 2023-12-02T02:59:58.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1900128256 bytes 2023-12-02T02:59:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 1972158464 bytes 2023-12-02T02:59:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2021181440 bytes 2023-12-02T02:59:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2055862272 bytes 2023-12-02T02:59:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2078559232 bytes 2023-12-02T02:59:58.591 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2087556096 bytes 2023-12-02T02:59:58.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2108109824 bytes 2023-12-02T02:59:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2164603904 bytes 2023-12-02T02:59:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2196005888 bytes 2023-12-02T02:59:59.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2262111232 bytes 2023-12-02T02:59:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2325448704 bytes 2023-12-02T02:59:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2337547264 bytes 2023-12-02T02:59:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2367275008 bytes 2023-12-02T02:59:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2438857728 bytes 2023-12-02T02:59:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2481757184 bytes 2023-12-02T03:00:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2538955776 bytes 2023-12-02T03:00:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2605319168 bytes 2023-12-02T03:00:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2631119872 bytes 2023-12-02T03:00:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2666304512 bytes 2023-12-02T03:00:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2711577600 bytes 2023-12-02T03:00:00.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2714846208 bytes 2023-12-02T03:00:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2751842304 bytes 2023-12-02T03:00:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 2798878720 bytes 2023-12-02T03:00:01.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 2850936832 bytes 2023-12-02T03:00:01.110 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 2877289472 bytes 2023-12-02T03:00:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 2898984960 bytes 2023-12-02T03:00:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 2969774080 bytes 2023-12-02T03:00:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3000760320 bytes 2023-12-02T03:00:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3012019200 bytes 2023-12-02T03:00:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3028909056 bytes 2023-12-02T03:00:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3055773696 bytes 2023-12-02T03:00:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3079092224 bytes 2023-12-02T03:00:01.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3111161856 bytes 2023-12-02T03:00:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3137610752 bytes 2023-12-02T03:00:02.108 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3138214912 bytes 2023-12-02T03:00:02.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8200 IOs, 3155926016 bytes 2023-12-02T03:00:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8300 IOs, 3191370752 bytes 2023-12-02T03:00:02.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8400 IOs, 3218917376 bytes 2023-12-02T03:00:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8500 IOs, 3267898368 bytes 2023-12-02T03:00:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8600 IOs, 3356879872 bytes 2023-12-02T03:00:02.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8700 IOs, 3394712576 bytes 2023-12-02T03:00:03.013 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-02T03:00:03.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-02T03:00:03.013 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-02T03:00:03.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-02T03:00:03.017 INFO:tasks.rbd_mirror_thrash:waiting for 72 secs before thrashing 2023-12-02T03:00:03.115 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8800 IOs, 3564897280 bytes 2023-12-02T03:00:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8900 IOs, 3630234624 bytes 2023-12-02T03:00:03.418 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9000 IOs, 3701766144 bytes 2023-12-02T03:00:03.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9100 IOs, 3718227968 bytes 2023-12-02T03:00:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:00:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:00:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:00:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:00:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:00:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_3 2023-12-02T03:00:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2023-12-02T03:00:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:00:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_3 2023-12-02T03:00:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:00:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:00:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_3 128M 2023-12-02T03:00:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:00:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_3 2023-12-02T03:00:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:00:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:00:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:00:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_3 2023-12-02T03:00:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:00:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:00:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:00:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2023-12-02T03:00:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:00:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_3 2023-12-02T03:00:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:00:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2023-12-02T03:00:04.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:00:04.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2023-12-02T03:00:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_3': 2023-12-02T03:00:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:00:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:00:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:00:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12a23ad7ddcd 2023-12-02T03:00:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12a23ad7ddcd 2023-12-02T03:00:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:00:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:00:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:00:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:00:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:00:03 2023 2023-12-02T03:00:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:00:03 2023 2023-12-02T03:00:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:00:03 2023 2023-12-02T03:00:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:00:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:00:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: a35ab54d-17a7-446d-984d-02da30173d31 2023-12-02T03:00:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:00:04.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:00:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_3 2023-12-02T03:00:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:00:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_3 2023-12-02T03:00:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:00:04.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=9 2023-12-02T03:00:04.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:00:04.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:00:04.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 11788288 bytes 2023-12-02T03:00:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 28313600 bytes 2023-12-02T03:00:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 28944384 bytes 2023-12-02T03:00:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 60305408 bytes 2023-12-02T03:00:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 120925184 bytes 2023-12-02T03:00:05.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 203778048 bytes 2023-12-02T03:00:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 238610432 bytes 2023-12-02T03:00:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 288685056 bytes 2023-12-02T03:00:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 341545984 bytes 2023-12-02T03:00:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 403919872 bytes 2023-12-02T03:00:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 479069184 bytes 2023-12-02T03:00:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 525899776 bytes 2023-12-02T03:00:06.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 627656704 bytes 2023-12-02T03:00:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 717079552 bytes 2023-12-02T03:00:06.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 771106816 bytes 2023-12-02T03:00:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 790419456 bytes 2023-12-02T03:00:06.993 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 835475456 bytes 2023-12-02T03:00:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 903349248 bytes 2023-12-02T03:00:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 932323328 bytes 2023-12-02T03:00:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 987356160 bytes 2023-12-02T03:00:07.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 1089572864 bytes 2023-12-02T03:00:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 1129951232 bytes 2023-12-02T03:00:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 1145317376 bytes 2023-12-02T03:00:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1187408896 bytes 2023-12-02T03:00:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1241695232 bytes 2023-12-02T03:00:08.178 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1294271488 bytes 2023-12-02T03:00:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1338419200 bytes 2023-12-02T03:00:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1387203584 bytes 2023-12-02T03:00:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1427741696 bytes 2023-12-02T03:00:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1444318208 bytes 2023-12-02T03:00:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1444908032 bytes 2023-12-02T03:00:08.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1472716800 bytes 2023-12-02T03:00:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1539119104 bytes 2023-12-02T03:00:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1557026816 bytes 2023-12-02T03:00:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1570868224 bytes 2023-12-02T03:00:09.210 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1574853632 bytes 2023-12-02T03:00:09.302 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1589827584 bytes 2023-12-02T03:00:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1660428288 bytes 2023-12-02T03:00:09.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1721844736 bytes 2023-12-02T03:00:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1722449920 bytes 2023-12-02T03:00:09.757 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1742080000 bytes 2023-12-02T03:00:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1791435776 bytes 2023-12-02T03:00:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1914607616 bytes 2023-12-02T03:00:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 2001368064 bytes 2023-12-02T03:00:10.467 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 2079208448 bytes 2023-12-02T03:00:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 2130830336 bytes 2023-12-02T03:00:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 2156291072 bytes 2023-12-02T03:00:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2179204096 bytes 2023-12-02T03:00:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2197110784 bytes 2023-12-02T03:00:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2224245760 bytes 2023-12-02T03:00:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2243901440 bytes 2023-12-02T03:00:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2349091840 bytes 2023-12-02T03:00:11.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2391767040 bytes 2023-12-02T03:00:11.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2427545600 bytes 2023-12-02T03:00:11.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2503400448 bytes 2023-12-02T03:00:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2531425280 bytes 2023-12-02T03:00:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2573034496 bytes 2023-12-02T03:00:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2624650240 bytes 2023-12-02T03:00:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2642109440 bytes 2023-12-02T03:00:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2725660672 bytes 2023-12-02T03:00:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2771220480 bytes 2023-12-02T03:00:12.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2791444480 bytes 2023-12-02T03:00:12.609 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2797648896 bytes 2023-12-02T03:00:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2875392000 bytes 2023-12-02T03:00:12.875 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2896380928 bytes 2023-12-02T03:00:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2909664256 bytes 2023-12-02T03:00:13.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2993229824 bytes 2023-12-02T03:00:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 3040754688 bytes 2023-12-02T03:00:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 3088800768 bytes 2023-12-02T03:00:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 3183144960 bytes 2023-12-02T03:00:13.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:00:13.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:00:13.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:00:13.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:00:13.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:00:13.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_4 2023-12-02T03:00:13.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2023-12-02T03:00:13.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:00:13.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:13.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:13.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:13.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_4 2023-12-02T03:00:13.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:13.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:00:13.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:00:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_4 128M 2023-12-02T03:00:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:00:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_4 2023-12-02T03:00:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:00:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:00:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:00:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_4 2023-12-02T03:00:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:00:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:00:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:00:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2023-12-02T03:00:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:00:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_4 2023-12-02T03:00:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:00:13.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2023-12-02T03:00:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:00:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2023-12-02T03:00:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_4': 2023-12-02T03:00:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:00:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:00:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:00:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12ae44b802d3 2023-12-02T03:00:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12ae44b802d3 2023-12-02T03:00:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:00:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:00:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:00:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:00:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:00:13 2023 2023-12-02T03:00:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:00:13 2023 2023-12-02T03:00:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:00:13 2023 2023-12-02T03:00:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:00:14.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:00:14.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 730af9f3-f6b6-4cf2-9474-5684c31f584d 2023-12-02T03:00:14.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:00:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:00:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_4 2023-12-02T03:00:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:00:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_4 2023-12-02T03:00:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:00:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=11 2023-12-02T03:00:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:00:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:00:14.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 15881216 bytes 2023-12-02T03:00:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 43017216 bytes 2023-12-02T03:00:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 43623424 bytes 2023-12-02T03:00:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 108090368 bytes 2023-12-02T03:00:15.317 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 171600896 bytes 2023-12-02T03:00:15.414 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 197809152 bytes 2023-12-02T03:00:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 246435840 bytes 2023-12-02T03:00:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 267151360 bytes 2023-12-02T03:00:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 292065280 bytes 2023-12-02T03:00:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 398113792 bytes 2023-12-02T03:00:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 469660672 bytes 2023-12-02T03:00:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 503278592 bytes 2023-12-02T03:00:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 527252480 bytes 2023-12-02T03:00:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 555283456 bytes 2023-12-02T03:00:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 624510976 bytes 2023-12-02T03:00:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 694386688 bytes 2023-12-02T03:00:16.791 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 695008256 bytes 2023-12-02T03:00:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 715059200 bytes 2023-12-02T03:00:17.069 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 815975424 bytes 2023-12-02T03:00:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 843156480 bytes 2023-12-02T03:00:17.276 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 881842176 bytes 2023-12-02T03:00:17.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 948335616 bytes 2023-12-02T03:00:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 965332992 bytes 2023-12-02T03:00:17.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1020755968 bytes 2023-12-02T03:00:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1021390848 bytes 2023-12-02T03:00:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1060381696 bytes 2023-12-02T03:00:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1080120320 bytes 2023-12-02T03:00:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1129786368 bytes 2023-12-02T03:00:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1153964032 bytes 2023-12-02T03:00:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1193695232 bytes 2023-12-02T03:00:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1227971584 bytes 2023-12-02T03:00:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1289099264 bytes 2023-12-02T03:00:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1343516672 bytes 2023-12-02T03:00:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1384354816 bytes 2023-12-02T03:00:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1446600704 bytes 2023-12-02T03:00:19.066 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1515998208 bytes 2023-12-02T03:00:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1525246976 bytes 2023-12-02T03:00:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1565315072 bytes 2023-12-02T03:00:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1573766144 bytes 2023-12-02T03:00:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1643230208 bytes 2023-12-02T03:00:19.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1670499328 bytes 2023-12-02T03:00:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1761034240 bytes 2023-12-02T03:00:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1813422080 bytes 2023-12-02T03:00:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1830947840 bytes 2023-12-02T03:00:20.211 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1889242112 bytes 2023-12-02T03:00:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1937745920 bytes 2023-12-02T03:00:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1956427776 bytes 2023-12-02T03:00:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2001796096 bytes 2023-12-02T03:00:20.692 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2053083136 bytes 2023-12-02T03:00:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2076420096 bytes 2023-12-02T03:00:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2145597440 bytes 2023-12-02T03:00:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2170178560 bytes 2023-12-02T03:00:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2206999552 bytes 2023-12-02T03:00:21.295 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2270038016 bytes 2023-12-02T03:00:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2283032576 bytes 2023-12-02T03:00:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2324780032 bytes 2023-12-02T03:00:21.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2408262656 bytes 2023-12-02T03:00:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2416585728 bytes 2023-12-02T03:00:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2468325376 bytes 2023-12-02T03:00:22.022 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2496481280 bytes 2023-12-02T03:00:22.136 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2535300096 bytes 2023-12-02T03:00:22.317 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2622836736 bytes 2023-12-02T03:00:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2646446080 bytes 2023-12-02T03:00:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2679416832 bytes 2023-12-02T03:00:22.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2713746432 bytes 2023-12-02T03:00:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2771607552 bytes 2023-12-02T03:00:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2810903552 bytes 2023-12-02T03:00:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2816826368 bytes 2023-12-02T03:00:23.124 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 2879160320 bytes 2023-12-02T03:00:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 2892983296 bytes 2023-12-02T03:00:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 2923317248 bytes 2023-12-02T03:00:23.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 2967596032 bytes 2023-12-02T03:00:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 2992175104 bytes 2023-12-02T03:00:23.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3023802368 bytes 2023-12-02T03:00:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3096035328 bytes 2023-12-02T03:00:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3146471424 bytes 2023-12-02T03:00:24.057 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3170842624 bytes 2023-12-02T03:00:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3240845312 bytes 2023-12-02T03:00:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3294785536 bytes 2023-12-02T03:00:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3321017344 bytes 2023-12-02T03:00:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3370398720 bytes 2023-12-02T03:00:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8200 IOs, 3393723392 bytes 2023-12-02T03:00:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8300 IOs, 3462621184 bytes 2023-12-02T03:00:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8400 IOs, 3542440960 bytes 2023-12-02T03:00:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8500 IOs, 3551219712 bytes 2023-12-02T03:00:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8600 IOs, 3578176512 bytes 2023-12-02T03:00:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8700 IOs, 3644841984 bytes 2023-12-02T03:00:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8800 IOs, 3728094208 bytes 2023-12-02T03:00:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:00:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:00:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:00:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:00:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:00:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_5 2023-12-02T03:00:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2023-12-02T03:00:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:00:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_5 2023-12-02T03:00:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:00:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:00:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_5 128M 2023-12-02T03:00:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:00:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_5 2023-12-02T03:00:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:00:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:00:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:00:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_5 2023-12-02T03:00:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:00:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:00:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:00:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2023-12-02T03:00:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:00:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_5 2023-12-02T03:00:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:00:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2023-12-02T03:00:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:00:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2023-12-02T03:00:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_5': 2023-12-02T03:00:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:00:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:00:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:00:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12ba342a059e 2023-12-02T03:00:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12ba342a059e 2023-12-02T03:00:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:00:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:00:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:00:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:00:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:00:25 2023 2023-12-02T03:00:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:00:25 2023 2023-12-02T03:00:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:00:25 2023 2023-12-02T03:00:26.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:00:26.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:00:26.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 19ecaa0e-982b-435c-a3a7-d881b478cee6 2023-12-02T03:00:26.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:00:26.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:00:26.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_5 2023-12-02T03:00:26.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:00:26.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:26.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_5 2023-12-02T03:00:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:00:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=7 2023-12-02T03:00:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:00:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:00:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 28984320 bytes 2023-12-02T03:00:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 103233536 bytes 2023-12-02T03:00:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 168626176 bytes 2023-12-02T03:00:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 206341120 bytes 2023-12-02T03:00:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 225165312 bytes 2023-12-02T03:00:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 289436672 bytes 2023-12-02T03:00:27.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 351041536 bytes 2023-12-02T03:00:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 353410048 bytes 2023-12-02T03:00:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 394129408 bytes 2023-12-02T03:00:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 414963712 bytes 2023-12-02T03:00:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 454620160 bytes 2023-12-02T03:00:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 457327616 bytes 2023-12-02T03:00:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 502481920 bytes 2023-12-02T03:00:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 552711168 bytes 2023-12-02T03:00:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 663142400 bytes 2023-12-02T03:00:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 721302528 bytes 2023-12-02T03:00:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 810705920 bytes 2023-12-02T03:00:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 815171584 bytes 2023-12-02T03:00:29.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 871252992 bytes 2023-12-02T03:00:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 909506560 bytes 2023-12-02T03:00:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 953248768 bytes 2023-12-02T03:00:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 1013629952 bytes 2023-12-02T03:00:29.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 1029603328 bytes 2023-12-02T03:00:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1081316352 bytes 2023-12-02T03:00:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1110620160 bytes 2023-12-02T03:00:29.888 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1172159488 bytes 2023-12-02T03:00:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1220948992 bytes 2023-12-02T03:00:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1303284736 bytes 2023-12-02T03:00:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1304422400 bytes 2023-12-02T03:00:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1338733568 bytes 2023-12-02T03:00:30.464 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1360366592 bytes 2023-12-02T03:00:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1397630976 bytes 2023-12-02T03:00:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1451092992 bytes 2023-12-02T03:00:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1462543360 bytes 2023-12-02T03:00:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1535310848 bytes 2023-12-02T03:00:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1579636736 bytes 2023-12-02T03:00:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1589985280 bytes 2023-12-02T03:00:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1644841984 bytes 2023-12-02T03:00:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1733408768 bytes 2023-12-02T03:00:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1793154048 bytes 2023-12-02T03:00:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1858338816 bytes 2023-12-02T03:00:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1865059328 bytes 2023-12-02T03:00:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1931160576 bytes 2023-12-02T03:00:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1975943168 bytes 2023-12-02T03:00:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1994565632 bytes 2023-12-02T03:00:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 2073034752 bytes 2023-12-02T03:00:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 2098074624 bytes 2023-12-02T03:00:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2168889344 bytes 2023-12-02T03:00:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2272277504 bytes 2023-12-02T03:00:33.150 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2337594368 bytes 2023-12-02T03:00:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2379000832 bytes 2023-12-02T03:00:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2519036928 bytes 2023-12-02T03:00:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:00:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:00:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:00:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:00:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:00:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_6 2023-12-02T03:00:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2023-12-02T03:00:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:00:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_6 2023-12-02T03:00:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:00:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:00:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_6 128M 2023-12-02T03:00:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:00:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_6 2023-12-02T03:00:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:00:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:00:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:00:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_6 2023-12-02T03:00:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:00:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:00:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:00:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2023-12-02T03:00:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:00:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_6 2023-12-02T03:00:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:00:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2023-12-02T03:00:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:00:34.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2023-12-02T03:00:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_6': 2023-12-02T03:00:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:00:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:00:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:00:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12c6462eeb2e 2023-12-02T03:00:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12c6462eeb2e 2023-12-02T03:00:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:00:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:00:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:00:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:00:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:00:33 2023 2023-12-02T03:00:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:00:33 2023 2023-12-02T03:00:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:00:33 2023 2023-12-02T03:00:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:00:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:00:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: f1a9b266-fc3d-4246-8a57-fe1695af4b4d 2023-12-02T03:00:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:00:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:00:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_6 2023-12-02T03:00:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:00:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_6 2023-12-02T03:00:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:00:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=10 2023-12-02T03:00:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:00:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:00:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 7271424 bytes 2023-12-02T03:00:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 8676352 bytes 2023-12-02T03:00:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 37678080 bytes 2023-12-02T03:00:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 55817216 bytes 2023-12-02T03:00:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 119614464 bytes 2023-12-02T03:00:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 172727296 bytes 2023-12-02T03:00:35.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 257566720 bytes 2023-12-02T03:00:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 346526720 bytes 2023-12-02T03:00:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 422218752 bytes 2023-12-02T03:00:35.811 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 450600960 bytes 2023-12-02T03:00:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 501463040 bytes 2023-12-02T03:00:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 502082560 bytes 2023-12-02T03:00:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 537556992 bytes 2023-12-02T03:00:36.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 585182208 bytes 2023-12-02T03:00:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 607060992 bytes 2023-12-02T03:00:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 641257472 bytes 2023-12-02T03:00:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 711568384 bytes 2023-12-02T03:00:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 780441600 bytes 2023-12-02T03:00:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 813975552 bytes 2023-12-02T03:00:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 835265536 bytes 2023-12-02T03:00:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 891615232 bytes 2023-12-02T03:00:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 922894336 bytes 2023-12-02T03:00:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 947367936 bytes 2023-12-02T03:00:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 984304640 bytes 2023-12-02T03:00:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1044938752 bytes 2023-12-02T03:00:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1120589824 bytes 2023-12-02T03:00:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1173915648 bytes 2023-12-02T03:00:38.100 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1247631360 bytes 2023-12-02T03:00:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1311748096 bytes 2023-12-02T03:00:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1359063040 bytes 2023-12-02T03:00:38.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1415022592 bytes 2023-12-02T03:00:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1434360832 bytes 2023-12-02T03:00:38.695 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1445603328 bytes 2023-12-02T03:00:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1488431104 bytes 2023-12-02T03:00:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1510472704 bytes 2023-12-02T03:00:39.094 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1586479104 bytes 2023-12-02T03:00:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1653249024 bytes 2023-12-02T03:00:39.334 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1670528000 bytes 2023-12-02T03:00:39.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1719134208 bytes 2023-12-02T03:00:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1762642944 bytes 2023-12-02T03:00:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1788285952 bytes 2023-12-02T03:00:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1841982464 bytes 2023-12-02T03:00:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1863790592 bytes 2023-12-02T03:00:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1923818496 bytes 2023-12-02T03:00:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1960677376 bytes 2023-12-02T03:00:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1964080128 bytes 2023-12-02T03:00:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1973970944 bytes 2023-12-02T03:00:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2063798272 bytes 2023-12-02T03:00:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2156303360 bytes 2023-12-02T03:00:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2320361472 bytes 2023-12-02T03:00:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2320953344 bytes 2023-12-02T03:00:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2378556416 bytes 2023-12-02T03:00:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2437895168 bytes 2023-12-02T03:00:41.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2498089984 bytes 2023-12-02T03:00:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2512994304 bytes 2023-12-02T03:00:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2549842944 bytes 2023-12-02T03:00:41.854 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2596009984 bytes 2023-12-02T03:00:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2628808704 bytes 2023-12-02T03:00:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2693926912 bytes 2023-12-02T03:00:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2787135488 bytes 2023-12-02T03:00:42.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2787735552 bytes 2023-12-02T03:00:42.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2800250880 bytes 2023-12-02T03:00:42.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2842743808 bytes 2023-12-02T03:00:42.693 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2855920640 bytes 2023-12-02T03:00:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2856767488 bytes 2023-12-02T03:00:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2943674368 bytes 2023-12-02T03:00:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2960538624 bytes 2023-12-02T03:00:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2975738880 bytes 2023-12-02T03:00:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 3056747520 bytes 2023-12-02T03:00:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 3102866432 bytes 2023-12-02T03:00:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 3195910144 bytes 2023-12-02T03:00:43.800 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 3272250368 bytes 2023-12-02T03:00:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 3333720064 bytes 2023-12-02T03:00:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3334325248 bytes 2023-12-02T03:00:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3430200320 bytes 2023-12-02T03:00:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3496875008 bytes 2023-12-02T03:00:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3517756416 bytes 2023-12-02T03:00:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:00:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:00:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:00:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:00:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:00:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_7 2023-12-02T03:00:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2023-12-02T03:00:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:00:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_7 2023-12-02T03:00:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:00:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:00:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_7 128M 2023-12-02T03:00:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:00:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_7 2023-12-02T03:00:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:00:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:00:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:00:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_7 2023-12-02T03:00:44.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:00:44.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:00:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:00:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2023-12-02T03:00:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:00:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_7 2023-12-02T03:00:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:00:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2023-12-02T03:00:45.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:00:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2023-12-02T03:00:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_7': 2023-12-02T03:00:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:00:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:00:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:00:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12d2c081ba6e 2023-12-02T03:00:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12d2c081ba6e 2023-12-02T03:00:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:00:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:00:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:00:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:00:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:00:44 2023 2023-12-02T03:00:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:00:44 2023 2023-12-02T03:00:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:00:44 2023 2023-12-02T03:00:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:00:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:00:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 14b5094b-8df2-410f-b1dd-ae91594e9848 2023-12-02T03:00:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:00:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:00:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_7 2023-12-02T03:00:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:00:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_7 2023-12-02T03:00:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:00:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=10 2023-12-02T03:00:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:00:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:00:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 71261184 bytes 2023-12-02T03:00:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 106456064 bytes 2023-12-02T03:00:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 154223616 bytes 2023-12-02T03:00:46.099 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 164807680 bytes 2023-12-02T03:00:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 177827840 bytes 2023-12-02T03:00:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 196954112 bytes 2023-12-02T03:00:46.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 254973952 bytes 2023-12-02T03:00:46.544 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 305144832 bytes 2023-12-02T03:00:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 369514496 bytes 2023-12-02T03:00:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 435281920 bytes 2023-12-02T03:00:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 558940160 bytes 2023-12-02T03:00:47.182 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 595914752 bytes 2023-12-02T03:00:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 600203264 bytes 2023-12-02T03:00:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 621939712 bytes 2023-12-02T03:00:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 677376000 bytes 2023-12-02T03:00:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 722822144 bytes 2023-12-02T03:00:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 781842432 bytes 2023-12-02T03:00:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 833264640 bytes 2023-12-02T03:00:48.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 939108352 bytes 2023-12-02T03:00:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 941451264 bytes 2023-12-02T03:00:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 942069760 bytes 2023-12-02T03:00:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 948198400 bytes 2023-12-02T03:00:48.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 959810560 bytes 2023-12-02T03:00:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1025191936 bytes 2023-12-02T03:00:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1082801152 bytes 2023-12-02T03:00:48.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1177011200 bytes 2023-12-02T03:00:49.062 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1233266688 bytes 2023-12-02T03:00:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1274825728 bytes 2023-12-02T03:00:49.320 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1321374720 bytes 2023-12-02T03:00:49.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1350346752 bytes 2023-12-02T03:00:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1351104512 bytes 2023-12-02T03:00:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1373012992 bytes 2023-12-02T03:00:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1386858496 bytes 2023-12-02T03:00:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1387468800 bytes 2023-12-02T03:00:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1406804992 bytes 2023-12-02T03:00:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1459526656 bytes 2023-12-02T03:00:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1486034944 bytes 2023-12-02T03:00:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1520322560 bytes 2023-12-02T03:00:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1544818688 bytes 2023-12-02T03:00:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1571908608 bytes 2023-12-02T03:00:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1598473216 bytes 2023-12-02T03:00:50.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1634609152 bytes 2023-12-02T03:00:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1733568512 bytes 2023-12-02T03:00:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1805220864 bytes 2023-12-02T03:00:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1872796672 bytes 2023-12-02T03:00:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1953790976 bytes 2023-12-02T03:00:51.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1984307200 bytes 2023-12-02T03:00:51.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2043558912 bytes 2023-12-02T03:00:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2064669696 bytes 2023-12-02T03:00:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2098151424 bytes 2023-12-02T03:00:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2113001472 bytes 2023-12-02T03:00:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2157805568 bytes 2023-12-02T03:00:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2192344064 bytes 2023-12-02T03:00:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2229655552 bytes 2023-12-02T03:00:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2230301696 bytes 2023-12-02T03:00:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2276318208 bytes 2023-12-02T03:00:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2315918336 bytes 2023-12-02T03:00:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2363954176 bytes 2023-12-02T03:00:52.836 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2375744512 bytes 2023-12-02T03:00:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2416260096 bytes 2023-12-02T03:00:53.110 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2488860672 bytes 2023-12-02T03:00:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2523435008 bytes 2023-12-02T03:00:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2571566080 bytes 2023-12-02T03:00:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2596627456 bytes 2023-12-02T03:00:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2651217920 bytes 2023-12-02T03:00:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2697065472 bytes 2023-12-02T03:00:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2737510400 bytes 2023-12-02T03:00:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2796085248 bytes 2023-12-02T03:00:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 2840757248 bytes 2023-12-02T03:00:54.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 2863577088 bytes 2023-12-02T03:00:54.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 2898057216 bytes 2023-12-02T03:00:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 2948312064 bytes 2023-12-02T03:00:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 2998987776 bytes 2023-12-02T03:00:54.859 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3076762624 bytes 2023-12-02T03:00:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3101341696 bytes 2023-12-02T03:00:55.086 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3139298304 bytes 2023-12-02T03:00:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3177260032 bytes 2023-12-02T03:00:55.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3226687488 bytes 2023-12-02T03:00:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3260899328 bytes 2023-12-02T03:00:55.550 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3266441216 bytes 2023-12-02T03:00:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:00:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:00:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:00:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:00:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:00:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_8 2023-12-02T03:00:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2023-12-02T03:00:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:00:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_8 2023-12-02T03:00:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:00:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:00:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_8 128M 2023-12-02T03:00:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:00:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_8 2023-12-02T03:00:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:00:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:00:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:00:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:00:55.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_8 2023-12-02T03:00:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:00:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:00:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:00:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2023-12-02T03:00:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:00:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_8 2023-12-02T03:00:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:00:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2023-12-02T03:00:56.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:00:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2023-12-02T03:00:56.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_8': 2023-12-02T03:00:56.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:00:56.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:00:56.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:00:56.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12def09e28e7 2023-12-02T03:00:56.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12def09e28e7 2023-12-02T03:00:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:00:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:00:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:00:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:00:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:00:55 2023 2023-12-02T03:00:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:00:55 2023 2023-12-02T03:00:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:00:55 2023 2023-12-02T03:00:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:00:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:00:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: f71e0710-9762-4b0b-8375-4f15e3c521f1 2023-12-02T03:00:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:00:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:00:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_8 2023-12-02T03:00:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:00:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:00:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_8 2023-12-02T03:00:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:00:56.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=14 2023-12-02T03:00:56.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:00:56.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:00:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 5687296 bytes 2023-12-02T03:00:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 47866880 bytes 2023-12-02T03:00:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 119337984 bytes 2023-12-02T03:00:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 147889152 bytes 2023-12-02T03:00:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 230716416 bytes 2023-12-02T03:00:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 268957696 bytes 2023-12-02T03:00:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 344465408 bytes 2023-12-02T03:00:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 398888960 bytes 2023-12-02T03:00:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 477889536 bytes 2023-12-02T03:00:57.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 502105088 bytes 2023-12-02T03:00:58.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 533816320 bytes 2023-12-02T03:00:58.176 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 580360192 bytes 2023-12-02T03:00:58.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 607280128 bytes 2023-12-02T03:00:58.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 680519680 bytes 2023-12-02T03:00:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 799839232 bytes 2023-12-02T03:00:58.822 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 853751808 bytes 2023-12-02T03:00:58.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 911117312 bytes 2023-12-02T03:00:59.123 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 974436352 bytes 2023-12-02T03:00:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 1009803264 bytes 2023-12-02T03:00:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 1042904064 bytes 2023-12-02T03:00:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 1061419008 bytes 2023-12-02T03:00:59.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 1122286592 bytes 2023-12-02T03:00:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 1151108096 bytes 2023-12-02T03:00:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1164334080 bytes 2023-12-02T03:00:59.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1176218624 bytes 2023-12-02T03:01:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1223116800 bytes 2023-12-02T03:01:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1286650880 bytes 2023-12-02T03:01:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1351387136 bytes 2023-12-02T03:01:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1431085056 bytes 2023-12-02T03:01:00.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1496766464 bytes 2023-12-02T03:01:00.797 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1540341760 bytes 2023-12-02T03:01:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1540958208 bytes 2023-12-02T03:01:01.003 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1602359296 bytes 2023-12-02T03:01:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1626168320 bytes 2023-12-02T03:01:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1679929344 bytes 2023-12-02T03:01:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1729968128 bytes 2023-12-02T03:01:01.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1788209152 bytes 2023-12-02T03:01:01.697 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1788817408 bytes 2023-12-02T03:01:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1828831232 bytes 2023-12-02T03:01:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1838471168 bytes 2023-12-02T03:01:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1846248448 bytes 2023-12-02T03:01:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1926484992 bytes 2023-12-02T03:01:02.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 2068932608 bytes 2023-12-02T03:01:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 2102437888 bytes 2023-12-02T03:01:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 2196464640 bytes 2023-12-02T03:01:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 2232495104 bytes 2023-12-02T03:01:03.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 2296711168 bytes 2023-12-02T03:01:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2378891264 bytes 2023-12-02T03:01:03.311 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2414939136 bytes 2023-12-02T03:01:03.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2452549632 bytes 2023-12-02T03:01:03.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2475430912 bytes 2023-12-02T03:01:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2504002560 bytes 2023-12-02T03:01:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2571580416 bytes 2023-12-02T03:01:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2602648576 bytes 2023-12-02T03:01:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2625978368 bytes 2023-12-02T03:01:04.144 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2656234496 bytes 2023-12-02T03:01:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2678307840 bytes 2023-12-02T03:01:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2690791424 bytes 2023-12-02T03:01:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2691378176 bytes 2023-12-02T03:01:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2754201600 bytes 2023-12-02T03:01:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2780627968 bytes 2023-12-02T03:01:04.809 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2808241152 bytes 2023-12-02T03:01:04.930 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2861761536 bytes 2023-12-02T03:01:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2890210304 bytes 2023-12-02T03:01:05.146 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2900960256 bytes 2023-12-02T03:01:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2915809280 bytes 2023-12-02T03:01:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2972370944 bytes 2023-12-02T03:01:05.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 3055890432 bytes 2023-12-02T03:01:05.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 3062147072 bytes 2023-12-02T03:01:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 3111305216 bytes 2023-12-02T03:01:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 3151825920 bytes 2023-12-02T03:01:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 3152432128 bytes 2023-12-02T03:01:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 3172772864 bytes 2023-12-02T03:01:06.186 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3189206016 bytes 2023-12-02T03:01:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3195848704 bytes 2023-12-02T03:01:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3242992640 bytes 2023-12-02T03:01:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3270315008 bytes 2023-12-02T03:01:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3354440704 bytes 2023-12-02T03:01:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3446547456 bytes 2023-12-02T03:01:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3523810304 bytes 2023-12-02T03:01:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3535177728 bytes 2023-12-02T03:01:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8200 IOs, 3577383936 bytes 2023-12-02T03:01:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8300 IOs, 3589984256 bytes 2023-12-02T03:01:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8400 IOs, 3633869824 bytes 2023-12-02T03:01:07.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8500 IOs, 3676227584 bytes 2023-12-02T03:01:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8600 IOs, 3725456384 bytes 2023-12-02T03:01:07.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8700 IOs, 3772966912 bytes 2023-12-02T03:01:08.026 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8800 IOs, 3805805568 bytes 2023-12-02T03:01:08.146 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8900 IOs, 3840907264 bytes 2023-12-02T03:01:08.235 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9000 IOs, 3854403584 bytes 2023-12-02T03:01:08.359 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9100 IOs, 3887788032 bytes 2023-12-02T03:01:08.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9200 IOs, 3952558080 bytes 2023-12-02T03:01:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9300 IOs, 4021844992 bytes 2023-12-02T03:01:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9400 IOs, 4062906368 bytes 2023-12-02T03:01:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9500 IOs, 4085326848 bytes 2023-12-02T03:01:09.027 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9600 IOs, 4110310400 bytes 2023-12-02T03:01:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9700 IOs, 4189356032 bytes 2023-12-02T03:01:09.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9800 IOs, 4228252672 bytes 2023-12-02T03:01:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9900 IOs, 4303105024 bytes 2023-12-02T03:01:09.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10000 IOs, 4347457536 bytes 2023-12-02T03:01:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10100 IOs, 4427869184 bytes 2023-12-02T03:01:09.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10200 IOs, 4503638016 bytes 2023-12-02T03:01:10.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10300 IOs, 4529179648 bytes 2023-12-02T03:01:10.228 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10400 IOs, 4585270272 bytes 2023-12-02T03:01:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10500 IOs, 4643979264 bytes 2023-12-02T03:01:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10600 IOs, 4682087424 bytes 2023-12-02T03:01:10.571 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:01:10.571 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:01:10.571 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:01:10.571 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:01:10.571 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:01:10.571 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_9 2023-12-02T03:01:10.571 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2023-12-02T03:01:10.571 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:01:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:01:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_9 2023-12-02T03:01:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:01:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:01:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_9 128M 2023-12-02T03:01:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:01:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:01:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_9 2023-12-02T03:01:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:01:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:01:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:01:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_9 2023-12-02T03:01:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:01:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:01:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:01:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:01:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2023-12-02T03:01:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:01:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:01:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_9 2023-12-02T03:01:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:01:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2023-12-02T03:01:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:01:11.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2023-12-02T03:01:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_9': 2023-12-02T03:01:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:01:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:01:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:01:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12ead6ff0558 2023-12-02T03:01:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12ead6ff0558 2023-12-02T03:01:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:01:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:01:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:01:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:01:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:01:10 2023 2023-12-02T03:01:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:01:10 2023 2023-12-02T03:01:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:01:10 2023 2023-12-02T03:01:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:01:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:01:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: da41f19a-67cd-4518-a939-22e8fe4b6971 2023-12-02T03:01:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:01:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:01:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_9 2023-12-02T03:01:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:01:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:01:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_9 2023-12-02T03:01:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:01:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=11 2023-12-02T03:01:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:01:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:01:11.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 35232768 bytes 2023-12-02T03:01:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 69861376 bytes 2023-12-02T03:01:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 106420224 bytes 2023-12-02T03:01:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 180136960 bytes 2023-12-02T03:01:12.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 240516096 bytes 2023-12-02T03:01:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 296910848 bytes 2023-12-02T03:01:12.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 307601408 bytes 2023-12-02T03:01:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 364597248 bytes 2023-12-02T03:01:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 446243840 bytes 2023-12-02T03:01:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 477568000 bytes 2023-12-02T03:01:13.105 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 569023488 bytes 2023-12-02T03:01:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 647769088 bytes 2023-12-02T03:01:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 684876800 bytes 2023-12-02T03:01:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 731370496 bytes 2023-12-02T03:01:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 731982848 bytes 2023-12-02T03:01:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 756594688 bytes 2023-12-02T03:01:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 896386048 bytes 2023-12-02T03:01:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 949910528 bytes 2023-12-02T03:01:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 1006765056 bytes 2023-12-02T03:01:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 1071720448 bytes 2023-12-02T03:01:14.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 1157789696 bytes 2023-12-02T03:01:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 1196286976 bytes 2023-12-02T03:01:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 1212909568 bytes 2023-12-02T03:01:15.017 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6947749278097154) > weight (0.25) 2023-12-02T03:01:15.017 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5004866669748962) > weight (0.25) 2023-12-02T03:01:15.018 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.859273649025363) > weight (0.25) 2023-12-02T03:01:15.018 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9935077959538621) > weight (0.25) 2023-12-02T03:01:15.018 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2023-12-02T03:01:15.058 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1297031168 bytes 2023-12-02T03:01:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1314863104 bytes 2023-12-02T03:01:15.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1339773952 bytes 2023-12-02T03:01:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1401359360 bytes 2023-12-02T03:01:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1435441152 bytes 2023-12-02T03:01:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1470610432 bytes 2023-12-02T03:01:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1553071104 bytes 2023-12-02T03:01:16.251 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1681815552 bytes 2023-12-02T03:01:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1847306240 bytes 2023-12-02T03:01:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1893626880 bytes 2023-12-02T03:01:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1978117120 bytes 2023-12-02T03:01:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 2028948480 bytes 2023-12-02T03:01:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 2073818112 bytes 2023-12-02T03:01:17.251 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 2108723200 bytes 2023-12-02T03:01:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 2161665024 bytes 2023-12-02T03:01:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 2166710272 bytes 2023-12-02T03:01:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 2216223744 bytes 2023-12-02T03:01:17.736 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 2248766464 bytes 2023-12-02T03:01:17.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 2272686080 bytes 2023-12-02T03:01:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 2294931456 bytes 2023-12-02T03:01:18.026 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-02T03:01:18.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-02T03:01:18.027 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-02T03:01:18.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr:2023-12-02T03:01:18.025+0000 7f04db111640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 98388) UID: 1000 2023-12-02T03:01:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 2295546880 bytes 2023-12-02T03:01:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 2375348224 bytes 2023-12-02T03:01:18.362 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 2410814464 bytes 2023-12-02T03:01:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 2438219776 bytes 2023-12-02T03:01:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2501143552 bytes 2023-12-02T03:01:18.778 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2559012864 bytes 2023-12-02T03:01:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2587090944 bytes 2023-12-02T03:01:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2615871488 bytes 2023-12-02T03:01:19.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2716999680 bytes 2023-12-02T03:01:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2740168704 bytes 2023-12-02T03:01:19.657 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2769519616 bytes 2023-12-02T03:01:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2812913664 bytes 2023-12-02T03:01:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2836810752 bytes 2023-12-02T03:01:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2843940864 bytes 2023-12-02T03:01:20.098 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2874536960 bytes 2023-12-02T03:01:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2934254592 bytes 2023-12-02T03:01:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 3024330752 bytes 2023-12-02T03:01:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 3100094464 bytes 2023-12-02T03:01:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 3145031680 bytes 2023-12-02T03:01:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 3201792000 bytes 2023-12-02T03:01:21.026 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 3244227584 bytes 2023-12-02T03:01:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 3307143168 bytes 2023-12-02T03:01:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 3346182144 bytes 2023-12-02T03:01:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 3356477440 bytes 2023-12-02T03:01:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 3414351872 bytes 2023-12-02T03:01:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 3444433920 bytes 2023-12-02T03:01:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 3464421376 bytes 2023-12-02T03:01:21.911 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 3513435136 bytes 2023-12-02T03:01:22.022 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 3541009408 bytes 2023-12-02T03:01:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 3541640192 bytes 2023-12-02T03:01:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3597444096 bytes 2023-12-02T03:01:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3606415360 bytes 2023-12-02T03:01:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3678872576 bytes 2023-12-02T03:01:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:01:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:01:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:01:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:01:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:01:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_10 2023-12-02T03:01:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2023-12-02T03:01:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:01:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:01:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_10 2023-12-02T03:01:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:01:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:01:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_10 128M 2023-12-02T03:01:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:01:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:01:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_10 2023-12-02T03:01:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:01:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:01:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:01:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_10 2023-12-02T03:01:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:01:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:01:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:01:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:01:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2023-12-02T03:01:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:01:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:01:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_10 2023-12-02T03:01:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:01:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2023-12-02T03:01:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:01:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2023-12-02T03:01:23.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_10': 2023-12-02T03:01:23.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:01:23.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:01:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:01:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12f5af8e31bc 2023-12-02T03:01:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12f5af8e31bc 2023-12-02T03:01:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:01:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:01:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:01:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:01:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:01:22 2023 2023-12-02T03:01:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:01:22 2023 2023-12-02T03:01:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:01:22 2023 2023-12-02T03:01:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:01:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:01:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: ddeae318-0d59-4bf2-b55e-a3f9934ab1b6 2023-12-02T03:01:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:01:23.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:01:23.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_10 2023-12-02T03:01:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:01:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:01:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_10 2023-12-02T03:01:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:01:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=12 2023-12-02T03:01:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:01:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:01:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 34910208 bytes 2023-12-02T03:01:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 75089920 bytes 2023-12-02T03:01:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 75687936 bytes 2023-12-02T03:01:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 193843200 bytes 2023-12-02T03:01:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 229508096 bytes 2023-12-02T03:01:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 258547712 bytes 2023-12-02T03:01:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 282870784 bytes 2023-12-02T03:01:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 343749632 bytes 2023-12-02T03:01:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 355461120 bytes 2023-12-02T03:01:24.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 384332800 bytes 2023-12-02T03:01:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 421710848 bytes 2023-12-02T03:01:25.027 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-02T03:01:25.027 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T03:01:25.028 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-02T03:01:25.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-02T03:01:25.028 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-02T03:01:25.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-02T03:01:25.031 INFO:tasks.rbd_mirror_thrash:waiting for 115 secs before thrashing 2023-12-02T03:01:25.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 431375360 bytes 2023-12-02T03:01:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 533412864 bytes 2023-12-02T03:01:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 603776000 bytes 2023-12-02T03:01:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 656337920 bytes 2023-12-02T03:01:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 704453632 bytes 2023-12-02T03:01:25.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 752846848 bytes 2023-12-02T03:01:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 784096256 bytes 2023-12-02T03:01:26.106 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 784678912 bytes 2023-12-02T03:01:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 869454848 bytes 2023-12-02T03:01:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 901870592 bytes 2023-12-02T03:01:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 919330816 bytes 2023-12-02T03:01:26.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 954616832 bytes 2023-12-02T03:01:26.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1039815680 bytes 2023-12-02T03:01:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1104584704 bytes 2023-12-02T03:01:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1154201600 bytes 2023-12-02T03:01:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1183027200 bytes 2023-12-02T03:01:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1225743360 bytes 2023-12-02T03:01:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1269563392 bytes 2023-12-02T03:01:27.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1331401728 bytes 2023-12-02T03:01:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1375471616 bytes 2023-12-02T03:01:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1401462784 bytes 2023-12-02T03:01:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1433816064 bytes 2023-12-02T03:01:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1462355968 bytes 2023-12-02T03:01:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1503561728 bytes 2023-12-02T03:01:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1627398144 bytes 2023-12-02T03:01:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1671588864 bytes 2023-12-02T03:01:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1672197120 bytes 2023-12-02T03:01:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1776564224 bytes 2023-12-02T03:01:28.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1798340608 bytes 2023-12-02T03:01:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1891625984 bytes 2023-12-02T03:01:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1967238144 bytes 2023-12-02T03:01:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 2010523648 bytes 2023-12-02T03:01:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 2065064960 bytes 2023-12-02T03:01:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 2129033216 bytes 2023-12-02T03:01:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 2129640448 bytes 2023-12-02T03:01:29.979 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 2174752768 bytes 2023-12-02T03:01:30.273 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2200428544 bytes 2023-12-02T03:01:30.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2291447808 bytes 2023-12-02T03:01:30.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2325875712 bytes 2023-12-02T03:01:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2351894528 bytes 2023-12-02T03:01:30.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2406550528 bytes 2023-12-02T03:01:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2472108032 bytes 2023-12-02T03:01:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2608246784 bytes 2023-12-02T03:01:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2622252032 bytes 2023-12-02T03:01:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2663060480 bytes 2023-12-02T03:01:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2729238528 bytes 2023-12-02T03:01:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2816546816 bytes 2023-12-02T03:01:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2888875008 bytes 2023-12-02T03:01:32.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2908529664 bytes 2023-12-02T03:01:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2941455360 bytes 2023-12-02T03:01:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2947712000 bytes 2023-12-02T03:01:32.411 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2986425344 bytes 2023-12-02T03:01:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2998317056 bytes 2023-12-02T03:01:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 3036486656 bytes 2023-12-02T03:01:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 3037751296 bytes 2023-12-02T03:01:32.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 3070694400 bytes 2023-12-02T03:01:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 3144310784 bytes 2023-12-02T03:01:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 3205252096 bytes 2023-12-02T03:01:33.260 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 3229011968 bytes 2023-12-02T03:01:33.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 3274778624 bytes 2023-12-02T03:01:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 3361664000 bytes 2023-12-02T03:01:33.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 3375412224 bytes 2023-12-02T03:01:33.817 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3416538112 bytes 2023-12-02T03:01:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3434774528 bytes 2023-12-02T03:01:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3502286848 bytes 2023-12-02T03:01:34.215 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3552748544 bytes 2023-12-02T03:01:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3553368064 bytes 2023-12-02T03:01:34.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3618841600 bytes 2023-12-02T03:01:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3639352320 bytes 2023-12-02T03:01:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3730228224 bytes 2023-12-02T03:01:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8200 IOs, 3795871744 bytes 2023-12-02T03:01:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8300 IOs, 3856883712 bytes 2023-12-02T03:01:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8400 IOs, 3866711040 bytes 2023-12-02T03:01:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8500 IOs, 3895810048 bytes 2023-12-02T03:01:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8600 IOs, 3914687488 bytes 2023-12-02T03:01:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8700 IOs, 3924462592 bytes 2023-12-02T03:01:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8800 IOs, 3927643136 bytes 2023-12-02T03:01:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:01:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:01:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:01:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:01:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:01:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_11 2023-12-02T03:01:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2023-12-02T03:01:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:01:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:01:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_11 2023-12-02T03:01:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:01:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:01:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_11 128M 2023-12-02T03:01:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:01:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:01:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_11 2023-12-02T03:01:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:01:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:01:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:01:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_11 2023-12-02T03:01:35.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:01:35.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:01:35.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:01:35.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:01:35.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2023-12-02T03:01:35.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:01:35.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:01:35.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_11 2023-12-02T03:01:35.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:01:35.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2023-12-02T03:01:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:01:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2023-12-02T03:01:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_11': 2023-12-02T03:01:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:01:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:01:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:01:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 1307cf84dba0 2023-12-02T03:01:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.1307cf84dba0 2023-12-02T03:01:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:01:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:01:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:01:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:01:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:01:35 2023 2023-12-02T03:01:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:01:35 2023 2023-12-02T03:01:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:01:35 2023 2023-12-02T03:01:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:01:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:01:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 258c1b03-48a5-42f2-8745-ac0283c8bc87 2023-12-02T03:01:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:01:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:01:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_11 2023-12-02T03:01:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:01:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:01:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_11 2023-12-02T03:01:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:01:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=14 2023-12-02T03:01:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:01:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:01:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 53519360 bytes 2023-12-02T03:01:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 63319040 bytes 2023-12-02T03:01:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 126965760 bytes 2023-12-02T03:01:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 144151552 bytes 2023-12-02T03:01:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 184932352 bytes 2023-12-02T03:01:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 189535232 bytes 2023-12-02T03:01:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 213117952 bytes 2023-12-02T03:01:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 240064512 bytes 2023-12-02T03:01:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 249475072 bytes 2023-12-02T03:01:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 264029184 bytes 2023-12-02T03:01:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 313252864 bytes 2023-12-02T03:01:37.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 358135808 bytes 2023-12-02T03:01:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 372818944 bytes 2023-12-02T03:01:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 438852608 bytes 2023-12-02T03:01:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 466434048 bytes 2023-12-02T03:01:38.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 518571008 bytes 2023-12-02T03:01:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 543235072 bytes 2023-12-02T03:01:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 568081408 bytes 2023-12-02T03:01:38.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 568686592 bytes 2023-12-02T03:01:38.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 619754496 bytes 2023-12-02T03:01:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 663579648 bytes 2023-12-02T03:01:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 682843136 bytes 2023-12-02T03:01:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 728526848 bytes 2023-12-02T03:01:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 744528896 bytes 2023-12-02T03:01:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 786488320 bytes 2023-12-02T03:01:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 806962176 bytes 2023-12-02T03:01:39.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 841668608 bytes 2023-12-02T03:01:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 881093632 bytes 2023-12-02T03:01:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 904529920 bytes 2023-12-02T03:01:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 941945856 bytes 2023-12-02T03:01:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 989089792 bytes 2023-12-02T03:01:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1005611008 bytes 2023-12-02T03:01:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1093655552 bytes 2023-12-02T03:01:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1161747456 bytes 2023-12-02T03:01:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1191122944 bytes 2023-12-02T03:01:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1200960512 bytes 2023-12-02T03:01:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1257672704 bytes 2023-12-02T03:01:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1273896960 bytes 2023-12-02T03:01:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1279819776 bytes 2023-12-02T03:01:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1359373312 bytes 2023-12-02T03:01:41.563 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1378465792 bytes 2023-12-02T03:01:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1379840000 bytes 2023-12-02T03:01:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1380451328 bytes 2023-12-02T03:01:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1410971648 bytes 2023-12-02T03:01:41.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1442697216 bytes 2023-12-02T03:01:42.098 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1482643456 bytes 2023-12-02T03:01:42.182 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1488074752 bytes 2023-12-02T03:01:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 1491929088 bytes 2023-12-02T03:01:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 1503625216 bytes 2023-12-02T03:01:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 1547282432 bytes 2023-12-02T03:01:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 1586899968 bytes 2023-12-02T03:01:42.771 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 1655143424 bytes 2023-12-02T03:01:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 1683246080 bytes 2023-12-02T03:01:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 1712380928 bytes 2023-12-02T03:01:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 1757688832 bytes 2023-12-02T03:01:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 1808168960 bytes 2023-12-02T03:01:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 1907155968 bytes 2023-12-02T03:01:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 1984479232 bytes 2023-12-02T03:01:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2009964544 bytes 2023-12-02T03:01:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2055801856 bytes 2023-12-02T03:01:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2104653824 bytes 2023-12-02T03:01:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2128996352 bytes 2023-12-02T03:01:44.337 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2176032768 bytes 2023-12-02T03:01:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2190245888 bytes 2023-12-02T03:01:44.553 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2223875072 bytes 2023-12-02T03:01:44.691 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2268856320 bytes 2023-12-02T03:01:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2312309760 bytes 2023-12-02T03:01:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2327589888 bytes 2023-12-02T03:01:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 2370022400 bytes 2023-12-02T03:01:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 2394524672 bytes 2023-12-02T03:01:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 2421443584 bytes 2023-12-02T03:01:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 2470460416 bytes 2023-12-02T03:01:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 2559382528 bytes 2023-12-02T03:01:45.781 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 2615677952 bytes 2023-12-02T03:01:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 2668616704 bytes 2023-12-02T03:01:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 2725932032 bytes 2023-12-02T03:01:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 2756959232 bytes 2023-12-02T03:01:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 2795664384 bytes 2023-12-02T03:01:46.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 2796299264 bytes 2023-12-02T03:01:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 2862538752 bytes 2023-12-02T03:01:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 2931608576 bytes 2023-12-02T03:01:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8200 IOs, 3032702976 bytes 2023-12-02T03:01:47.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8300 IOs, 3071002624 bytes 2023-12-02T03:01:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8400 IOs, 3099078656 bytes 2023-12-02T03:01:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8500 IOs, 3145809920 bytes 2023-12-02T03:01:47.524 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8600 IOs, 3240755200 bytes 2023-12-02T03:01:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8700 IOs, 3274804224 bytes 2023-12-02T03:01:47.781 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8800 IOs, 3298197504 bytes 2023-12-02T03:01:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8900 IOs, 3303881728 bytes 2023-12-02T03:01:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9000 IOs, 3334678528 bytes 2023-12-02T03:01:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9100 IOs, 3405406208 bytes 2023-12-02T03:01:48.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9200 IOs, 3470473216 bytes 2023-12-02T03:01:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9300 IOs, 3522387968 bytes 2023-12-02T03:01:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9400 IOs, 3546073088 bytes 2023-12-02T03:01:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9500 IOs, 3568631808 bytes 2023-12-02T03:01:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9600 IOs, 3605380096 bytes 2023-12-02T03:01:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9700 IOs, 3627216896 bytes 2023-12-02T03:01:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9800 IOs, 3656758272 bytes 2023-12-02T03:01:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9900 IOs, 3746376704 bytes 2023-12-02T03:01:49.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10000 IOs, 3813168128 bytes 2023-12-02T03:01:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10100 IOs, 3885790208 bytes 2023-12-02T03:01:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10200 IOs, 3927556096 bytes 2023-12-02T03:01:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10300 IOs, 3969241088 bytes 2023-12-02T03:01:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10400 IOs, 4035480576 bytes 2023-12-02T03:01:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10500 IOs, 4114796544 bytes 2023-12-02T03:01:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10600 IOs, 4171625472 bytes 2023-12-02T03:01:50.482 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10700 IOs, 4208780288 bytes 2023-12-02T03:01:50.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:01:50.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:01:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:01:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:01:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:01:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_12 2023-12-02T03:01:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2023-12-02T03:01:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:01:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:01:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_12 2023-12-02T03:01:50.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:50.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:01:50.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:01:50.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:50.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_12 128M 2023-12-02T03:01:50.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:01:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:01:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_12 2023-12-02T03:01:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:01:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:01:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:01:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:01:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_12 2023-12-02T03:01:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:01:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:01:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:01:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:01:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2023-12-02T03:01:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:01:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:01:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_12 2023-12-02T03:01:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:01:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2023-12-02T03:01:51.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:01:51.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2023-12-02T03:01:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_12': 2023-12-02T03:01:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:01:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:01:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:01:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 1311e4b07246 2023-12-02T03:01:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.1311e4b07246 2023-12-02T03:01:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:01:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:01:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:01:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:01:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:01:50 2023 2023-12-02T03:01:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:01:50 2023 2023-12-02T03:01:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:01:50 2023 2023-12-02T03:01:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:01:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:01:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: a30b9861-e056-4151-bcfe-e4a4f2e867dd 2023-12-02T03:01:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:01:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:01:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_12 2023-12-02T03:01:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:01:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:01:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_12 2023-12-02T03:01:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:01:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=9 2023-12-02T03:01:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:01:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:01:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 16692224 bytes 2023-12-02T03:01:51.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 80399360 bytes 2023-12-02T03:01:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 92584960 bytes 2023-12-02T03:01:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 124916736 bytes 2023-12-02T03:01:52.243 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 163721216 bytes 2023-12-02T03:01:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 182700032 bytes 2023-12-02T03:01:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 244235264 bytes 2023-12-02T03:01:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 316152832 bytes 2023-12-02T03:01:52.749 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 327664640 bytes 2023-12-02T03:01:52.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 382739456 bytes 2023-12-02T03:01:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 401681408 bytes 2023-12-02T03:01:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 416100352 bytes 2023-12-02T03:01:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 460876800 bytes 2023-12-02T03:01:53.370 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 518217728 bytes 2023-12-02T03:01:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 532373504 bytes 2023-12-02T03:01:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 532981760 bytes 2023-12-02T03:01:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 555621376 bytes 2023-12-02T03:01:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 586261504 bytes 2023-12-02T03:01:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 621880320 bytes 2023-12-02T03:01:54.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 674840576 bytes 2023-12-02T03:01:54.162 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 706311168 bytes 2023-12-02T03:01:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 766981120 bytes 2023-12-02T03:01:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 809062400 bytes 2023-12-02T03:01:54.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 822272000 bytes 2023-12-02T03:01:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 863140864 bytes 2023-12-02T03:01:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 934784000 bytes 2023-12-02T03:01:55.009 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 997220352 bytes 2023-12-02T03:01:55.122 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1025099776 bytes 2023-12-02T03:01:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1083812864 bytes 2023-12-02T03:01:55.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1196893184 bytes 2023-12-02T03:01:55.603 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1213060096 bytes 2023-12-02T03:01:55.747 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1266566144 bytes 2023-12-02T03:01:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1297309696 bytes 2023-12-02T03:01:55.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1327285248 bytes 2023-12-02T03:01:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1435823104 bytes 2023-12-02T03:01:56.310 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1460604928 bytes 2023-12-02T03:01:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1526381568 bytes 2023-12-02T03:01:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1541538816 bytes 2023-12-02T03:01:56.729 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1574367232 bytes 2023-12-02T03:01:56.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1574985728 bytes 2023-12-02T03:01:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1602807808 bytes 2023-12-02T03:01:57.071 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1662540800 bytes 2023-12-02T03:01:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1667723264 bytes 2023-12-02T03:01:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1728788480 bytes 2023-12-02T03:01:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1769334784 bytes 2023-12-02T03:01:57.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1797152768 bytes 2023-12-02T03:01:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1875417088 bytes 2023-12-02T03:01:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 1935755264 bytes 2023-12-02T03:01:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 1971286016 bytes 2023-12-02T03:01:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2058747904 bytes 2023-12-02T03:01:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2078060544 bytes 2023-12-02T03:01:58.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2086468608 bytes 2023-12-02T03:01:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2099538944 bytes 2023-12-02T03:01:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2159009792 bytes 2023-12-02T03:01:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2178406400 bytes 2023-12-02T03:01:59.038 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2309514240 bytes 2023-12-02T03:01:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2378352640 bytes 2023-12-02T03:01:59.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2422064128 bytes 2023-12-02T03:01:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2466570240 bytes 2023-12-02T03:01:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2487103488 bytes 2023-12-02T03:01:59.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2512393216 bytes 2023-12-02T03:01:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2555407360 bytes 2023-12-02T03:01:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2580535296 bytes 2023-12-02T03:02:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2666563584 bytes 2023-12-02T03:02:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2679768064 bytes 2023-12-02T03:02:00.467 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2724254720 bytes 2023-12-02T03:02:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2758879232 bytes 2023-12-02T03:02:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:02:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:02:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:02:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:02:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:02:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_13 2023-12-02T03:02:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2023-12-02T03:02:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:02:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_13 2023-12-02T03:02:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:02:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:02:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_13 128M 2023-12-02T03:02:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:02:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_13 2023-12-02T03:02:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:02:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:02:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:02:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_13 2023-12-02T03:02:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:02:00.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:00.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:02:00.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:02:00.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2023-12-02T03:02:00.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:02:00.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:00.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_13 2023-12-02T03:02:00.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:02:00.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2023-12-02T03:02:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:02:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2023-12-02T03:02:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_13': 2023-12-02T03:02:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:02:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:02:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:02:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 131953c6b45d 2023-12-02T03:02:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.131953c6b45d 2023-12-02T03:02:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:02:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:02:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:02:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:02:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:02:00 2023 2023-12-02T03:02:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:02:00 2023 2023-12-02T03:02:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:02:00 2023 2023-12-02T03:02:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:02:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:02:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: cdf57a21-3ee2-466a-9e94-c4dd4500ddd1 2023-12-02T03:02:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:02:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:02:01.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_13 2023-12-02T03:02:01.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:02:01.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:01.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_13 2023-12-02T03:02:01.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:02:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=12 2023-12-02T03:02:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:02:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:02:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 49878016 bytes 2023-12-02T03:02:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 133921792 bytes 2023-12-02T03:02:02.107 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 224450560 bytes 2023-12-02T03:02:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 245339136 bytes 2023-12-02T03:02:02.321 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 257809408 bytes 2023-12-02T03:02:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 349919232 bytes 2023-12-02T03:02:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 392609792 bytes 2023-12-02T03:02:02.795 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 460891136 bytes 2023-12-02T03:02:02.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 518170624 bytes 2023-12-02T03:02:03.067 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 552676352 bytes 2023-12-02T03:02:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 639322112 bytes 2023-12-02T03:02:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 678635520 bytes 2023-12-02T03:02:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 695303168 bytes 2023-12-02T03:02:03.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 734106624 bytes 2023-12-02T03:02:03.706 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 741353472 bytes 2023-12-02T03:02:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 765116416 bytes 2023-12-02T03:02:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 818375680 bytes 2023-12-02T03:02:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 850113536 bytes 2023-12-02T03:02:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 850735104 bytes 2023-12-02T03:02:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 876059648 bytes 2023-12-02T03:02:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 909402112 bytes 2023-12-02T03:02:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 945984512 bytes 2023-12-02T03:02:04.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 995819520 bytes 2023-12-02T03:02:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1069468672 bytes 2023-12-02T03:02:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1070056448 bytes 2023-12-02T03:02:04.984 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1073881088 bytes 2023-12-02T03:02:05.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1106930688 bytes 2023-12-02T03:02:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1127691264 bytes 2023-12-02T03:02:05.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1132914688 bytes 2023-12-02T03:02:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1138570240 bytes 2023-12-02T03:02:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1157664768 bytes 2023-12-02T03:02:05.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1193466880 bytes 2023-12-02T03:02:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1235543040 bytes 2023-12-02T03:02:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1322244096 bytes 2023-12-02T03:02:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1360269312 bytes 2023-12-02T03:02:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1365391360 bytes 2023-12-02T03:02:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1392958464 bytes 2023-12-02T03:02:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1412280320 bytes 2023-12-02T03:02:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1446307840 bytes 2023-12-02T03:02:06.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1465689088 bytes 2023-12-02T03:02:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1492746240 bytes 2023-12-02T03:02:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1554266112 bytes 2023-12-02T03:02:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1554896896 bytes 2023-12-02T03:02:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1610896384 bytes 2023-12-02T03:02:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1645169664 bytes 2023-12-02T03:02:07.370 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1687090176 bytes 2023-12-02T03:02:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1687711744 bytes 2023-12-02T03:02:07.563 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 1712965632 bytes 2023-12-02T03:02:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 1778643968 bytes 2023-12-02T03:02:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 1823417344 bytes 2023-12-02T03:02:07.981 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 1867265024 bytes 2023-12-02T03:02:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 1885556736 bytes 2023-12-02T03:02:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 1909349376 bytes 2023-12-02T03:02:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 1912970240 bytes 2023-12-02T03:02:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 1949404160 bytes 2023-12-02T03:02:08.530 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 1995212800 bytes 2023-12-02T03:02:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2024517632 bytes 2023-12-02T03:02:08.780 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2077844480 bytes 2023-12-02T03:02:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2151241728 bytes 2023-12-02T03:02:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2189241344 bytes 2023-12-02T03:02:09.230 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2235984896 bytes 2023-12-02T03:02:09.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2312537088 bytes 2023-12-02T03:02:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2368816128 bytes 2023-12-02T03:02:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2431803392 bytes 2023-12-02T03:02:09.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2467485696 bytes 2023-12-02T03:02:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2507755520 bytes 2023-12-02T03:02:10.062 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2527830016 bytes 2023-12-02T03:02:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2591703040 bytes 2023-12-02T03:02:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 2642139136 bytes 2023-12-02T03:02:10.464 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 2667295744 bytes 2023-12-02T03:02:10.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 2698441728 bytes 2023-12-02T03:02:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 2719575040 bytes 2023-12-02T03:02:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 2742532096 bytes 2023-12-02T03:02:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 2796538880 bytes 2023-12-02T03:02:11.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 2840625152 bytes 2023-12-02T03:02:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 2873515008 bytes 2023-12-02T03:02:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 2912653312 bytes 2023-12-02T03:02:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 2930190336 bytes 2023-12-02T03:02:11.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 2972336128 bytes 2023-12-02T03:02:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 2989764608 bytes 2023-12-02T03:02:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3001558016 bytes 2023-12-02T03:02:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8200 IOs, 3046507520 bytes 2023-12-02T03:02:12.202 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8300 IOs, 3072237568 bytes 2023-12-02T03:02:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8400 IOs, 3128459264 bytes 2023-12-02T03:02:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8500 IOs, 3173310464 bytes 2023-12-02T03:02:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8600 IOs, 3253617664 bytes 2023-12-02T03:02:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8700 IOs, 3327437824 bytes 2023-12-02T03:02:12.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8800 IOs, 3363476480 bytes 2023-12-02T03:02:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8900 IOs, 3424939008 bytes 2023-12-02T03:02:13.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9000 IOs, 3485777920 bytes 2023-12-02T03:02:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9100 IOs, 3539993600 bytes 2023-12-02T03:02:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9200 IOs, 3591343104 bytes 2023-12-02T03:02:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:02:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:02:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:02:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:02:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:02:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_14 2023-12-02T03:02:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2023-12-02T03:02:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:02:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_14 2023-12-02T03:02:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:02:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:02:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_14 128M 2023-12-02T03:02:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:02:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_14 2023-12-02T03:02:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:13.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:02:13.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:02:13.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:02:13.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:13.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_14 2023-12-02T03:02:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:02:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:02:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:02:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2023-12-02T03:02:13.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:02:13.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:13.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_14 2023-12-02T03:02:13.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:02:13.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2023-12-02T03:02:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:02:14.120 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2023-12-02T03:02:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_14': 2023-12-02T03:02:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:02:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:02:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:02:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 1323450110fc 2023-12-02T03:02:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.1323450110fc 2023-12-02T03:02:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:02:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:02:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:02:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:02:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:02:13 2023 2023-12-02T03:02:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:02:13 2023 2023-12-02T03:02:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:02:13 2023 2023-12-02T03:02:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:02:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:02:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 1d600ef5-05fa-4ec8-bb36-7d891860097c 2023-12-02T03:02:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:02:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:02:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_14 2023-12-02T03:02:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:02:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_14 2023-12-02T03:02:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:02:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=10 2023-12-02T03:02:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:02:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:02:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 26999808 bytes 2023-12-02T03:02:14.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 61382656 bytes 2023-12-02T03:02:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 117400576 bytes 2023-12-02T03:02:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 187977728 bytes 2023-12-02T03:02:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 204497920 bytes 2023-12-02T03:02:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 252636160 bytes 2023-12-02T03:02:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 307520512 bytes 2023-12-02T03:02:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 340681728 bytes 2023-12-02T03:02:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 353774592 bytes 2023-12-02T03:02:15.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 393771008 bytes 2023-12-02T03:02:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 443782144 bytes 2023-12-02T03:02:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 485658624 bytes 2023-12-02T03:02:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 534943744 bytes 2023-12-02T03:02:15.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 577934336 bytes 2023-12-02T03:02:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 611209216 bytes 2023-12-02T03:02:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 660021248 bytes 2023-12-02T03:02:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 680718336 bytes 2023-12-02T03:02:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 712646656 bytes 2023-12-02T03:02:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 746472448 bytes 2023-12-02T03:02:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 762555392 bytes 2023-12-02T03:02:16.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 818039808 bytes 2023-12-02T03:02:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 843683840 bytes 2023-12-02T03:02:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 907699200 bytes 2023-12-02T03:02:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 917947392 bytes 2023-12-02T03:02:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 952356864 bytes 2023-12-02T03:02:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 956526592 bytes 2023-12-02T03:02:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1012556800 bytes 2023-12-02T03:02:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1021342720 bytes 2023-12-02T03:02:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1054361600 bytes 2023-12-02T03:02:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1109791744 bytes 2023-12-02T03:02:18.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1156101120 bytes 2023-12-02T03:02:18.227 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1191041024 bytes 2023-12-02T03:02:18.310 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1192592384 bytes 2023-12-02T03:02:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1225660416 bytes 2023-12-02T03:02:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1247138816 bytes 2023-12-02T03:02:18.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1272355840 bytes 2023-12-02T03:02:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1362522112 bytes 2023-12-02T03:02:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1391694848 bytes 2023-12-02T03:02:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1411140608 bytes 2023-12-02T03:02:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1437967360 bytes 2023-12-02T03:02:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1476573184 bytes 2023-12-02T03:02:19.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1515515904 bytes 2023-12-02T03:02:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1560460288 bytes 2023-12-02T03:02:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1577977856 bytes 2023-12-02T03:02:19.800 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1631983616 bytes 2023-12-02T03:02:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1649753088 bytes 2023-12-02T03:02:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1691346944 bytes 2023-12-02T03:02:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 1716800512 bytes 2023-12-02T03:02:20.327 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 1784276992 bytes 2023-12-02T03:02:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 1811923968 bytes 2023-12-02T03:02:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 1820578816 bytes 2023-12-02T03:02:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 1872165888 bytes 2023-12-02T03:02:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 1916694528 bytes 2023-12-02T03:02:20.938 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 1969830912 bytes 2023-12-02T03:02:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2032325632 bytes 2023-12-02T03:02:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2071094272 bytes 2023-12-02T03:02:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2171328512 bytes 2023-12-02T03:02:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2214936576 bytes 2023-12-02T03:02:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2237741056 bytes 2023-12-02T03:02:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2278948864 bytes 2023-12-02T03:02:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2350114816 bytes 2023-12-02T03:02:22.137 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2390136832 bytes 2023-12-02T03:02:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2447714304 bytes 2023-12-02T03:02:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2495026176 bytes 2023-12-02T03:02:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2557719552 bytes 2023-12-02T03:02:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2571736064 bytes 2023-12-02T03:02:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2679052288 bytes 2023-12-02T03:02:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2704316416 bytes 2023-12-02T03:02:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 2704924672 bytes 2023-12-02T03:02:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 2743731200 bytes 2023-12-02T03:02:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 2772751360 bytes 2023-12-02T03:02:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 2792326144 bytes 2023-12-02T03:02:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 2816114688 bytes 2023-12-02T03:02:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 2831360000 bytes 2023-12-02T03:02:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 2877742080 bytes 2023-12-02T03:02:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 2918202368 bytes 2023-12-02T03:02:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 2971465728 bytes 2023-12-02T03:02:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:02:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:02:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:02:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:02:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:02:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_15 2023-12-02T03:02:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2023-12-02T03:02:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:02:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_15 2023-12-02T03:02:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:02:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:02:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_15 128M 2023-12-02T03:02:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:02:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_15 2023-12-02T03:02:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:02:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:02:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:02:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_15 2023-12-02T03:02:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:02:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:02:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:02:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2023-12-02T03:02:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:02:24.218 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:24.218 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_15 2023-12-02T03:02:24.218 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:02:24.218 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2023-12-02T03:02:24.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:02:24.535 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2023-12-02T03:02:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_15': 2023-12-02T03:02:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:02:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:02:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:02:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 132d9b7688c5 2023-12-02T03:02:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.132d9b7688c5 2023-12-02T03:02:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:02:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:02:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:02:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:02:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:02:24 2023 2023-12-02T03:02:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:02:24 2023 2023-12-02T03:02:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:02:24 2023 2023-12-02T03:02:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:02:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:02:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 5a34854e-65bf-43d7-9d2c-e0481cc5a159 2023-12-02T03:02:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:02:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:02:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_15 2023-12-02T03:02:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:02:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_15 2023-12-02T03:02:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:02:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=10 2023-12-02T03:02:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:02:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:02:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 94783488 bytes 2023-12-02T03:02:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 126649344 bytes 2023-12-02T03:02:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 185580544 bytes 2023-12-02T03:02:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 232927232 bytes 2023-12-02T03:02:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 280070144 bytes 2023-12-02T03:02:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 345761792 bytes 2023-12-02T03:02:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 370651136 bytes 2023-12-02T03:02:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 412262400 bytes 2023-12-02T03:02:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 446755840 bytes 2023-12-02T03:02:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 461628416 bytes 2023-12-02T03:02:26.019 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7666029144375885) > weight (0.25) 2023-12-02T03:02:26.019 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-02T03:02:26.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-02T03:02:26.020 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-02T03:02:26.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi057.stderr:2023-12-02T03:02:26.018+0000 7f71a82ca640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 93455) UID: 1000 2023-12-02T03:02:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 502545408 bytes 2023-12-02T03:02:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 504587264 bytes 2023-12-02T03:02:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 505203712 bytes 2023-12-02T03:02:26.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 520876032 bytes 2023-12-02T03:02:26.523 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 559148032 bytes 2023-12-02T03:02:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 602256384 bytes 2023-12-02T03:02:26.853 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 684787712 bytes 2023-12-02T03:02:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 742325248 bytes 2023-12-02T03:02:27.020 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-02T03:02:27.020 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T03:02:27.021 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-02T03:02:27.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-02T03:02:27.021 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-02T03:02:27.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-02T03:02:27.025 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2023-12-02T03:02:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 765486080 bytes 2023-12-02T03:02:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 804618240 bytes 2023-12-02T03:02:27.364 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 811468800 bytes 2023-12-02T03:02:27.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 893035520 bytes 2023-12-02T03:02:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 903528448 bytes 2023-12-02T03:02:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 925638656 bytes 2023-12-02T03:02:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1001783296 bytes 2023-12-02T03:02:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1068523520 bytes 2023-12-02T03:02:28.235 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1131315200 bytes 2023-12-02T03:02:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1137504256 bytes 2023-12-02T03:02:28.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1159538688 bytes 2023-12-02T03:02:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1160161280 bytes 2023-12-02T03:02:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1224527872 bytes 2023-12-02T03:02:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1264435200 bytes 2023-12-02T03:02:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1298351104 bytes 2023-12-02T03:02:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1336845312 bytes 2023-12-02T03:02:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1341759488 bytes 2023-12-02T03:02:29.252 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1362160640 bytes 2023-12-02T03:02:29.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1402466304 bytes 2023-12-02T03:02:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1452966912 bytes 2023-12-02T03:02:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1482373120 bytes 2023-12-02T03:02:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1498101760 bytes 2023-12-02T03:02:29.981 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1615434752 bytes 2023-12-02T03:02:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1666810880 bytes 2023-12-02T03:02:30.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1721649152 bytes 2023-12-02T03:02:30.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1740082176 bytes 2023-12-02T03:02:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1766471680 bytes 2023-12-02T03:02:30.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1798584320 bytes 2023-12-02T03:02:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1799196672 bytes 2023-12-02T03:02:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 1848684544 bytes 2023-12-02T03:02:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 1883033600 bytes 2023-12-02T03:02:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 1994022912 bytes 2023-12-02T03:02:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2015964160 bytes 2023-12-02T03:02:31.485 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2129108992 bytes 2023-12-02T03:02:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2150445056 bytes 2023-12-02T03:02:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2192674816 bytes 2023-12-02T03:02:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2200284160 bytes 2023-12-02T03:02:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2237037568 bytes 2023-12-02T03:02:32.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2283570176 bytes 2023-12-02T03:02:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2370182144 bytes 2023-12-02T03:02:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2391489536 bytes 2023-12-02T03:02:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2501386240 bytes 2023-12-02T03:02:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2549423104 bytes 2023-12-02T03:02:32.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2608053248 bytes 2023-12-02T03:02:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2656406528 bytes 2023-12-02T03:02:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2691932160 bytes 2023-12-02T03:02:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2722786304 bytes 2023-12-02T03:02:33.442 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2737989632 bytes 2023-12-02T03:02:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2761808896 bytes 2023-12-02T03:02:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2875262976 bytes 2023-12-02T03:02:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 2927020032 bytes 2023-12-02T03:02:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 2991732736 bytes 2023-12-02T03:02:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 3031303168 bytes 2023-12-02T03:02:34.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 3064805376 bytes 2023-12-02T03:02:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 3082961920 bytes 2023-12-02T03:02:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3083575296 bytes 2023-12-02T03:02:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:02:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:02:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:02:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:02:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:02:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_16 2023-12-02T03:02:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2023-12-02T03:02:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:02:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_16 2023-12-02T03:02:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:02:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:02:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_16 128M 2023-12-02T03:02:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:02:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_16 2023-12-02T03:02:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:02:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:02:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:02:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_16 2023-12-02T03:02:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:02:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:02:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:02:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2023-12-02T03:02:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:02:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_16 2023-12-02T03:02:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:02:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2023-12-02T03:02:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:02:35.558 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2023-12-02T03:02:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_16': 2023-12-02T03:02:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:02:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:02:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:02:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 133c95cd2034 2023-12-02T03:02:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.133c95cd2034 2023-12-02T03:02:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:02:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:02:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:02:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:02:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:02:34 2023 2023-12-02T03:02:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:02:34 2023 2023-12-02T03:02:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:02:34 2023 2023-12-02T03:02:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:02:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:02:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 807f0ce9-67ef-4e3a-987e-bfe34f97a832 2023-12-02T03:02:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:02:35.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:02:35.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_16 2023-12-02T03:02:35.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:02:35.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:35.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_16 2023-12-02T03:02:35.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:02:35.603 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=13 2023-12-02T03:02:35.603 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:02:35.603 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:02:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 87087104 bytes 2023-12-02T03:02:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 102517760 bytes 2023-12-02T03:02:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 177516544 bytes 2023-12-02T03:02:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 220549120 bytes 2023-12-02T03:02:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 262947840 bytes 2023-12-02T03:02:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 311146496 bytes 2023-12-02T03:02:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 319686656 bytes 2023-12-02T03:02:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 341485568 bytes 2023-12-02T03:02:36.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 393623552 bytes 2023-12-02T03:02:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 420510720 bytes 2023-12-02T03:02:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 456758272 bytes 2023-12-02T03:02:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 479368192 bytes 2023-12-02T03:02:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 510930944 bytes 2023-12-02T03:02:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 541403136 bytes 2023-12-02T03:02:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 582465536 bytes 2023-12-02T03:02:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 612767744 bytes 2023-12-02T03:02:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 622956544 bytes 2023-12-02T03:02:37.872 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 640031744 bytes 2023-12-02T03:02:37.986 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 666524672 bytes 2023-12-02T03:02:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 707373056 bytes 2023-12-02T03:02:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 748859392 bytes 2023-12-02T03:02:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 794999808 bytes 2023-12-02T03:02:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 820085760 bytes 2023-12-02T03:02:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 859494400 bytes 2023-12-02T03:02:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 893402112 bytes 2023-12-02T03:02:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 942763008 bytes 2023-12-02T03:02:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 971845632 bytes 2023-12-02T03:02:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1049518080 bytes 2023-12-02T03:02:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1081110528 bytes 2023-12-02T03:02:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1120613376 bytes 2023-12-02T03:02:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1126986752 bytes 2023-12-02T03:02:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1175121920 bytes 2023-12-02T03:02:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1189510144 bytes 2023-12-02T03:02:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1264730112 bytes 2023-12-02T03:02:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1296063488 bytes 2023-12-02T03:02:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1390177280 bytes 2023-12-02T03:02:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1402326016 bytes 2023-12-02T03:02:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1426366464 bytes 2023-12-02T03:02:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1479770112 bytes 2023-12-02T03:02:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1499244544 bytes 2023-12-02T03:02:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1558614016 bytes 2023-12-02T03:02:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1621392384 bytes 2023-12-02T03:02:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1654095872 bytes 2023-12-02T03:02:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1711096832 bytes 2023-12-02T03:02:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1713043456 bytes 2023-12-02T03:02:41.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1749210112 bytes 2023-12-02T03:02:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1749823488 bytes 2023-12-02T03:02:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 1787264000 bytes 2023-12-02T03:02:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 1792900096 bytes 2023-12-02T03:02:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 1850528768 bytes 2023-12-02T03:02:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 1886586880 bytes 2023-12-02T03:02:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 1950320640 bytes 2023-12-02T03:02:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 1956462592 bytes 2023-12-02T03:02:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 1971407872 bytes 2023-12-02T03:02:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 1994553344 bytes 2023-12-02T03:02:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2001839104 bytes 2023-12-02T03:02:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2058595328 bytes 2023-12-02T03:02:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2119246848 bytes 2023-12-02T03:02:43.255 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2175029248 bytes 2023-12-02T03:02:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2219550720 bytes 2023-12-02T03:02:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2286577664 bytes 2023-12-02T03:02:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2333937664 bytes 2023-12-02T03:02:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2406994944 bytes 2023-12-02T03:02:44.008 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2443652096 bytes 2023-12-02T03:02:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2523789312 bytes 2023-12-02T03:02:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2568671232 bytes 2023-12-02T03:02:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2612447232 bytes 2023-12-02T03:02:44.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2638624768 bytes 2023-12-02T03:02:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 2663379968 bytes 2023-12-02T03:02:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 2712881152 bytes 2023-12-02T03:02:44.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 2735815680 bytes 2023-12-02T03:02:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 2770923520 bytes 2023-12-02T03:02:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 2865251328 bytes 2023-12-02T03:02:45.413 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 2880077824 bytes 2023-12-02T03:02:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 2916310016 bytes 2023-12-02T03:02:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 2931956736 bytes 2023-12-02T03:02:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 2932602880 bytes 2023-12-02T03:02:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 2942271488 bytes 2023-12-02T03:02:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 2965851136 bytes 2023-12-02T03:02:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 2996141056 bytes 2023-12-02T03:02:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3086955520 bytes 2023-12-02T03:02:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8200 IOs, 3120036864 bytes 2023-12-02T03:02:46.544 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8300 IOs, 3157553152 bytes 2023-12-02T03:02:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8400 IOs, 3195325440 bytes 2023-12-02T03:02:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8500 IOs, 3260420096 bytes 2023-12-02T03:02:46.927 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8600 IOs, 3272904704 bytes 2023-12-02T03:02:47.097 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8700 IOs, 3346666496 bytes 2023-12-02T03:02:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8800 IOs, 3471534080 bytes 2023-12-02T03:02:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8900 IOs, 3492731904 bytes 2023-12-02T03:02:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9000 IOs, 3524127744 bytes 2023-12-02T03:02:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9100 IOs, 3555258368 bytes 2023-12-02T03:02:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9200 IOs, 3560532992 bytes 2023-12-02T03:02:47.867 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9300 IOs, 3561158656 bytes 2023-12-02T03:02:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9400 IOs, 3566040064 bytes 2023-12-02T03:02:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9500 IOs, 3657164800 bytes 2023-12-02T03:02:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9600 IOs, 3715164160 bytes 2023-12-02T03:02:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9700 IOs, 3823131648 bytes 2023-12-02T03:02:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:02:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:02:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:02:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:02:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:02:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_17 2023-12-02T03:02:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2023-12-02T03:02:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:02:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_17 2023-12-02T03:02:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:02:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:02:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_17 128M 2023-12-02T03:02:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:02:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_17 2023-12-02T03:02:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:02:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:02:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:02:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_17 2023-12-02T03:02:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:02:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:02:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:02:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2023-12-02T03:02:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:02:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_17 2023-12-02T03:02:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:02:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2023-12-02T03:02:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:02:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2023-12-02T03:02:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_17': 2023-12-02T03:02:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:02:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:02:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:02:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 1344ca26e7ee 2023-12-02T03:02:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.1344ca26e7ee 2023-12-02T03:02:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:02:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:02:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:02:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:02:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:02:48 2023 2023-12-02T03:02:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:02:48 2023 2023-12-02T03:02:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:02:48 2023 2023-12-02T03:02:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:02:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:02:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 1f4ea1ff-7710-42b2-9726-194a70ed8f1e 2023-12-02T03:02:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:02:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:02:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_17 2023-12-02T03:02:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:02:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_17 2023-12-02T03:02:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:02:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=8 2023-12-02T03:02:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:02:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:02:49.757 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 52968448 bytes 2023-12-02T03:02:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 112582656 bytes 2023-12-02T03:02:50.109 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 214387712 bytes 2023-12-02T03:02:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 237560832 bytes 2023-12-02T03:02:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 258887680 bytes 2023-12-02T03:02:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 282144768 bytes 2023-12-02T03:02:50.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 355511296 bytes 2023-12-02T03:02:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 388658176 bytes 2023-12-02T03:02:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 436742144 bytes 2023-12-02T03:02:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 532655104 bytes 2023-12-02T03:02:51.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 615951360 bytes 2023-12-02T03:02:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 645117952 bytes 2023-12-02T03:02:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 645743616 bytes 2023-12-02T03:02:51.599 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 678499328 bytes 2023-12-02T03:02:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 745638912 bytes 2023-12-02T03:02:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 834231296 bytes 2023-12-02T03:02:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 874374144 bytes 2023-12-02T03:02:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 874975232 bytes 2023-12-02T03:02:52.174 INFO:tasks.ceph.cluster2.osd.1.smithi057.stderr:2023-12-02T03:02:52.172+0000 7fbd9d384640 -1 reset not still connected to 0x55bdbe48e1a0 2023-12-02T03:02:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 893754368 bytes 2023-12-02T03:02:52.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 968581120 bytes 2023-12-02T03:02:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 1024351232 bytes 2023-12-02T03:02:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 1040387072 bytes 2023-12-02T03:02:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 1072583680 bytes 2023-12-02T03:02:52.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1095564288 bytes 2023-12-02T03:02:53.065 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1139995648 bytes 2023-12-02T03:02:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1189890048 bytes 2023-12-02T03:02:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1233944576 bytes 2023-12-02T03:02:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1241094144 bytes 2023-12-02T03:02:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1336657920 bytes 2023-12-02T03:02:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1365678080 bytes 2023-12-02T03:02:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1388097536 bytes 2023-12-02T03:02:54.006 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1431366656 bytes 2023-12-02T03:02:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1482739712 bytes 2023-12-02T03:02:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1542641664 bytes 2023-12-02T03:02:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1602198528 bytes 2023-12-02T03:02:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1656514560 bytes 2023-12-02T03:02:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1711115264 bytes 2023-12-02T03:02:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1782944768 bytes 2023-12-02T03:02:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1839822848 bytes 2023-12-02T03:02:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1875930112 bytes 2023-12-02T03:02:55.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1876520960 bytes 2023-12-02T03:02:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1900255232 bytes 2023-12-02T03:02:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1991051264 bytes 2023-12-02T03:02:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 2007483392 bytes 2023-12-02T03:02:55.873 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 2080550912 bytes 2023-12-02T03:02:56.061 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 2171776000 bytes 2023-12-02T03:02:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 2192154624 bytes 2023-12-02T03:02:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2250903552 bytes 2023-12-02T03:02:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2281890816 bytes 2023-12-02T03:02:56.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2290838528 bytes 2023-12-02T03:02:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2357549056 bytes 2023-12-02T03:02:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2452085760 bytes 2023-12-02T03:02:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2476647424 bytes 2023-12-02T03:02:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2493230080 bytes 2023-12-02T03:02:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2515539968 bytes 2023-12-02T03:02:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2529697792 bytes 2023-12-02T03:02:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2546952192 bytes 2023-12-02T03:02:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:02:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:02:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:02:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:02:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:02:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_18 2023-12-02T03:02:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2023-12-02T03:02:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:02:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:57.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:57.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_18 2023-12-02T03:02:57.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:57.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:02:57.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:02:57.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:57.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_18 128M 2023-12-02T03:02:57.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:02:57.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:57.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:57.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:57.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_18 2023-12-02T03:02:57.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:57.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:02:57.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:02:57.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:02:57.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:02:57.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_18 2023-12-02T03:02:57.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:02:57.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:57.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:02:57.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:02:57.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2023-12-02T03:02:57.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:02:57.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:57.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_18 2023-12-02T03:02:57.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:02:57.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2023-12-02T03:02:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:02:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2023-12-02T03:02:58.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_18': 2023-12-02T03:02:58.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:02:58.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:02:58.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:02:58.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 1350f9d650cb 2023-12-02T03:02:58.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.1350f9d650cb 2023-12-02T03:02:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:02:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:02:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:02:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:02:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:02:57 2023 2023-12-02T03:02:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:02:57 2023 2023-12-02T03:02:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:02:57 2023 2023-12-02T03:02:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:02:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:02:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: b8d1a79d-fa19-4b64-ba6b-5a4658335ba3 2023-12-02T03:02:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:02:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:02:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_18 2023-12-02T03:02:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:02:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:02:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_18 2023-12-02T03:02:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:02:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=12 2023-12-02T03:02:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:02:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:02:58.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 92737536 bytes 2023-12-02T03:02:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 140511232 bytes 2023-12-02T03:02:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 196114432 bytes 2023-12-02T03:02:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 267632640 bytes 2023-12-02T03:02:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 276449280 bytes 2023-12-02T03:02:59.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 306031616 bytes 2023-12-02T03:02:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 332151808 bytes 2023-12-02T03:02:59.766 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 380665856 bytes 2023-12-02T03:02:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 388485120 bytes 2023-12-02T03:02:59.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 413460480 bytes 2023-12-02T03:03:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 457534464 bytes 2023-12-02T03:03:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 491891712 bytes 2023-12-02T03:03:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 600003584 bytes 2023-12-02T03:03:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 621643776 bytes 2023-12-02T03:03:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 724257792 bytes 2023-12-02T03:03:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 777200640 bytes 2023-12-02T03:03:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 809412608 bytes 2023-12-02T03:03:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 810788864 bytes 2023-12-02T03:03:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 857120768 bytes 2023-12-02T03:03:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 857726976 bytes 2023-12-02T03:03:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 939068416 bytes 2023-12-02T03:03:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 958905344 bytes 2023-12-02T03:03:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 1036331008 bytes 2023-12-02T03:03:02.105 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1108911104 bytes 2023-12-02T03:03:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1112558592 bytes 2023-12-02T03:03:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1136689152 bytes 2023-12-02T03:03:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1201644544 bytes 2023-12-02T03:03:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1229332480 bytes 2023-12-02T03:03:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1289586688 bytes 2023-12-02T03:03:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1332915200 bytes 2023-12-02T03:03:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1376328704 bytes 2023-12-02T03:03:03.152 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1400989696 bytes 2023-12-02T03:03:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1408421888 bytes 2023-12-02T03:03:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1466409984 bytes 2023-12-02T03:03:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1495694336 bytes 2023-12-02T03:03:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1554315264 bytes 2023-12-02T03:03:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1583228928 bytes 2023-12-02T03:03:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1617940480 bytes 2023-12-02T03:03:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1644100608 bytes 2023-12-02T03:03:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1680008192 bytes 2023-12-02T03:03:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1743024128 bytes 2023-12-02T03:03:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1811878912 bytes 2023-12-02T03:03:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1833851904 bytes 2023-12-02T03:03:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1880098816 bytes 2023-12-02T03:03:04.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1930486784 bytes 2023-12-02T03:03:05.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1978895360 bytes 2023-12-02T03:03:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 2038018048 bytes 2023-12-02T03:03:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2051475456 bytes 2023-12-02T03:03:05.483 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2126445568 bytes 2023-12-02T03:03:05.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2164218880 bytes 2023-12-02T03:03:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2190120960 bytes 2023-12-02T03:03:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2216641536 bytes 2023-12-02T03:03:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2258682880 bytes 2023-12-02T03:03:06.135 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2306309120 bytes 2023-12-02T03:03:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2348389376 bytes 2023-12-02T03:03:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2388553728 bytes 2023-12-02T03:03:06.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2418743296 bytes 2023-12-02T03:03:06.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2459917312 bytes 2023-12-02T03:03:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2514624512 bytes 2023-12-02T03:03:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2560570368 bytes 2023-12-02T03:03:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2620115968 bytes 2023-12-02T03:03:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2623795200 bytes 2023-12-02T03:03:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2649593856 bytes 2023-12-02T03:03:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2716393472 bytes 2023-12-02T03:03:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2757429248 bytes 2023-12-02T03:03:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2843895808 bytes 2023-12-02T03:03:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2858434560 bytes 2023-12-02T03:03:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2881468416 bytes 2023-12-02T03:03:08.223 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 2969520128 bytes 2023-12-02T03:03:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 2975866880 bytes 2023-12-02T03:03:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 3015180288 bytes 2023-12-02T03:03:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 3035878400 bytes 2023-12-02T03:03:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 3087421440 bytes 2023-12-02T03:03:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3124671488 bytes 2023-12-02T03:03:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3223602176 bytes 2023-12-02T03:03:09.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3250731008 bytes 2023-12-02T03:03:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3329478656 bytes 2023-12-02T03:03:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3333747712 bytes 2023-12-02T03:03:09.778 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3391705088 bytes 2023-12-02T03:03:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3415403520 bytes 2023-12-02T03:03:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3452715008 bytes 2023-12-02T03:03:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8200 IOs, 3468722176 bytes 2023-12-02T03:03:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8300 IOs, 3548507136 bytes 2023-12-02T03:03:10.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8400 IOs, 3626785792 bytes 2023-12-02T03:03:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:03:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:03:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:03:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:03:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:03:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_19 2023-12-02T03:03:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2023-12-02T03:03:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:03:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_19 2023-12-02T03:03:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:03:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:03:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_19 128M 2023-12-02T03:03:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:03:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_19 2023-12-02T03:03:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:03:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:03:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:03:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_19 2023-12-02T03:03:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:03:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:03:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:03:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2023-12-02T03:03:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:03:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_19 2023-12-02T03:03:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:03:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2023-12-02T03:03:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:03:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2023-12-02T03:03:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_19': 2023-12-02T03:03:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:03:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:03:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:03:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 1359da788549 2023-12-02T03:03:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.1359da788549 2023-12-02T03:03:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:03:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:03:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:03:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:03:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:03:10 2023 2023-12-02T03:03:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:03:10 2023 2023-12-02T03:03:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:03:10 2023 2023-12-02T03:03:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:03:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:03:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: faedaa7c-e337-491d-80ec-3cfa45a9493c 2023-12-02T03:03:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:03:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:03:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_19 2023-12-02T03:03:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:03:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_19 2023-12-02T03:03:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:03:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=11 2023-12-02T03:03:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:03:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:03:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 23607296 bytes 2023-12-02T03:03:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 38287360 bytes 2023-12-02T03:03:11.955 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 38919168 bytes 2023-12-02T03:03:12.068 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 85108736 bytes 2023-12-02T03:03:12.237 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 140046336 bytes 2023-12-02T03:03:12.364 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 182981632 bytes 2023-12-02T03:03:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 205604864 bytes 2023-12-02T03:03:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 237411328 bytes 2023-12-02T03:03:12.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 283296768 bytes 2023-12-02T03:03:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 382886912 bytes 2023-12-02T03:03:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 402564096 bytes 2023-12-02T03:03:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 467057664 bytes 2023-12-02T03:03:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 527395840 bytes 2023-12-02T03:03:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 573147136 bytes 2023-12-02T03:03:13.692 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 633794560 bytes 2023-12-02T03:03:13.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 731536384 bytes 2023-12-02T03:03:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 747550720 bytes 2023-12-02T03:03:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 783133696 bytes 2023-12-02T03:03:14.245 INFO:tasks.ceph.cluster2.osd.2.smithi057.stderr:2023-12-02T03:03:14.243+0000 7f34767b9640 -1 reset not still connected to 0x56224d1c2270 2023-12-02T03:03:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 811810816 bytes 2023-12-02T03:03:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 849185792 bytes 2023-12-02T03:03:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 907590656 bytes 2023-12-02T03:03:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 950264832 bytes 2023-12-02T03:03:14.828 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 974685184 bytes 2023-12-02T03:03:14.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1010350080 bytes 2023-12-02T03:03:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1081756672 bytes 2023-12-02T03:03:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1160991744 bytes 2023-12-02T03:03:15.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1196069888 bytes 2023-12-02T03:03:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1213734912 bytes 2023-12-02T03:03:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1234968576 bytes 2023-12-02T03:03:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1271366656 bytes 2023-12-02T03:03:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1326242816 bytes 2023-12-02T03:03:16.147 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1403919360 bytes 2023-12-02T03:03:16.322 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1479316480 bytes 2023-12-02T03:03:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1503956992 bytes 2023-12-02T03:03:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1566110720 bytes 2023-12-02T03:03:16.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1626959872 bytes 2023-12-02T03:03:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1670828032 bytes 2023-12-02T03:03:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1699151872 bytes 2023-12-02T03:03:17.174 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1743560704 bytes 2023-12-02T03:03:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1760147456 bytes 2023-12-02T03:03:17.440 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1808442368 bytes 2023-12-02T03:03:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1847829504 bytes 2023-12-02T03:03:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1860933632 bytes 2023-12-02T03:03:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1881672704 bytes 2023-12-02T03:03:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1958497280 bytes 2023-12-02T03:03:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 2065015808 bytes 2023-12-02T03:03:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 2111749120 bytes 2023-12-02T03:03:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2158569472 bytes 2023-12-02T03:03:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2182363136 bytes 2023-12-02T03:03:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2239506432 bytes 2023-12-02T03:03:18.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2262091776 bytes 2023-12-02T03:03:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2344085504 bytes 2023-12-02T03:03:19.171 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2367309824 bytes 2023-12-02T03:03:19.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2375399424 bytes 2023-12-02T03:03:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2424705024 bytes 2023-12-02T03:03:19.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2486483968 bytes 2023-12-02T03:03:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2540386304 bytes 2023-12-02T03:03:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2574320640 bytes 2023-12-02T03:03:20.009 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2618832896 bytes 2023-12-02T03:03:20.031 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3936119740668472) > weight (0.25) 2023-12-02T03:03:20.031 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9593345532495302) > weight (0.25) 2023-12-02T03:03:20.031 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3013863203610454) > weight (0.25) 2023-12-02T03:03:20.031 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-12-02T03:03:20.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-12-02T03:03:20.032 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-02T03:03:20.032 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-12-02T03:03:20.032 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T03:03:20.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi057.stderr:2023-12-02T03:03:20.031+0000 7fe03cb30640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 95521) UID: 1000 2023-12-02T03:03:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2727564288 bytes 2023-12-02T03:03:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2728199168 bytes 2023-12-02T03:03:20.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2772611072 bytes 2023-12-02T03:03:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2800908288 bytes 2023-12-02T03:03:20.758 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2862301184 bytes 2023-12-02T03:03:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2900642816 bytes 2023-12-02T03:03:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2922808320 bytes 2023-12-02T03:03:21.105 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2936099840 bytes 2023-12-02T03:03:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2953604096 bytes 2023-12-02T03:03:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 3037661184 bytes 2023-12-02T03:03:21.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 3053223936 bytes 2023-12-02T03:03:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 3116704768 bytes 2023-12-02T03:03:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 3121623040 bytes 2023-12-02T03:03:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 3169176576 bytes 2023-12-02T03:03:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3206375424 bytes 2023-12-02T03:03:22.203 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3252833280 bytes 2023-12-02T03:03:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3307364352 bytes 2023-12-02T03:03:22.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3345631232 bytes 2023-12-02T03:03:22.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3388280832 bytes 2023-12-02T03:03:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:03:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:03:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:03:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:03:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:03:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_20 2023-12-02T03:03:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2023-12-02T03:03:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:03:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_20 2023-12-02T03:03:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:03:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:03:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_20 128M 2023-12-02T03:03:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:03:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_20 2023-12-02T03:03:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:03:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:03:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:03:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_20 2023-12-02T03:03:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:03:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:03:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:03:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2023-12-02T03:03:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:03:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_20 2023-12-02T03:03:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:03:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2023-12-02T03:03:23.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:03:23.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2023-12-02T03:03:23.591 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_20': 2023-12-02T03:03:23.591 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:03:23.591 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:03:23.591 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:03:23.591 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 13636741aade 2023-12-02T03:03:23.591 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.13636741aade 2023-12-02T03:03:23.591 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:03:23.591 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:03:23.591 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:03:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:03:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:03:22 2023 2023-12-02T03:03:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:03:22 2023 2023-12-02T03:03:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:03:22 2023 2023-12-02T03:03:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:03:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:03:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 8117a00a-84c1-4b06-96bd-60840c20cce4 2023-12-02T03:03:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:03:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:03:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_20 2023-12-02T03:03:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:03:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_20 2023-12-02T03:03:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:03:23.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=6 2023-12-02T03:03:23.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:03:23.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:03:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 91578368 bytes 2023-12-02T03:03:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 148506624 bytes 2023-12-02T03:03:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 224241664 bytes 2023-12-02T03:03:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 245275648 bytes 2023-12-02T03:03:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 271049728 bytes 2023-12-02T03:03:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 284028928 bytes 2023-12-02T03:03:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 374128640 bytes 2023-12-02T03:03:24.811 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 421711872 bytes 2023-12-02T03:03:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 455538688 bytes 2023-12-02T03:03:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 527619072 bytes 2023-12-02T03:03:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 575655936 bytes 2023-12-02T03:03:25.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 632820736 bytes 2023-12-02T03:03:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 653993984 bytes 2023-12-02T03:03:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 729373696 bytes 2023-12-02T03:03:25.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 747455488 bytes 2023-12-02T03:03:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 808057856 bytes 2023-12-02T03:03:26.134 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2023-12-02T03:03:26.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-12-02T03:03:26.135 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-12-02T03:03:26.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-12-02T03:03:26.138 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2023-12-02T03:03:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 863710208 bytes 2023-12-02T03:03:26.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 907970560 bytes 2023-12-02T03:03:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 944257024 bytes 2023-12-02T03:03:26.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 958053376 bytes 2023-12-02T03:03:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 972553216 bytes 2023-12-02T03:03:26.892 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 1014368256 bytes 2023-12-02T03:03:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 1045076992 bytes 2023-12-02T03:03:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1103175680 bytes 2023-12-02T03:03:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1194635264 bytes 2023-12-02T03:03:27.530 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1271879680 bytes 2023-12-02T03:03:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1314123776 bytes 2023-12-02T03:03:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1381033984 bytes 2023-12-02T03:03:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1451616256 bytes 2023-12-02T03:03:28.124 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1492624384 bytes 2023-12-02T03:03:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1508067328 bytes 2023-12-02T03:03:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1516149760 bytes 2023-12-02T03:03:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1542921216 bytes 2023-12-02T03:03:28.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1581543424 bytes 2023-12-02T03:03:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1624180736 bytes 2023-12-02T03:03:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1691600896 bytes 2023-12-02T03:03:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1774384128 bytes 2023-12-02T03:03:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1786631168 bytes 2023-12-02T03:03:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1836174336 bytes 2023-12-02T03:03:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1857011712 bytes 2023-12-02T03:03:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1898394624 bytes 2023-12-02T03:03:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:03:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:03:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:03:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:03:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:03:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_21 2023-12-02T03:03:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2023-12-02T03:03:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:03:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_21 2023-12-02T03:03:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:03:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:03:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_21 128M 2023-12-02T03:03:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:03:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_21 2023-12-02T03:03:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:03:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:03:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:03:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_21 2023-12-02T03:03:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:03:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:03:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:03:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2023-12-02T03:03:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:03:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_21 2023-12-02T03:03:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:03:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2023-12-02T03:03:30.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:03:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2023-12-02T03:03:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_21': 2023-12-02T03:03:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:03:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:03:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:03:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 137533ef857d 2023-12-02T03:03:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.137533ef857d 2023-12-02T03:03:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:03:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:03:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:03:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:03:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:03:29 2023 2023-12-02T03:03:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:03:29 2023 2023-12-02T03:03:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:03:29 2023 2023-12-02T03:03:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:03:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:03:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: b629353c-89f1-46c0-b5ff-f652e93d5cb3 2023-12-02T03:03:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:03:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:03:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_21 2023-12-02T03:03:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:03:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_21 2023-12-02T03:03:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:03:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=9 2023-12-02T03:03:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:03:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:03:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 7999488 bytes 2023-12-02T03:03:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 8622080 bytes 2023-12-02T03:03:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 22510592 bytes 2023-12-02T03:03:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 85709824 bytes 2023-12-02T03:03:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 141048832 bytes 2023-12-02T03:03:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 181715968 bytes 2023-12-02T03:03:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 227227648 bytes 2023-12-02T03:03:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 264543232 bytes 2023-12-02T03:03:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 303960064 bytes 2023-12-02T03:03:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 322384896 bytes 2023-12-02T03:03:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 377025536 bytes 2023-12-02T03:03:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 443673600 bytes 2023-12-02T03:03:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 476294144 bytes 2023-12-02T03:03:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 482882560 bytes 2023-12-02T03:03:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 499919872 bytes 2023-12-02T03:03:32.809 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 555703296 bytes 2023-12-02T03:03:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 556328960 bytes 2023-12-02T03:03:33.026 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.28627042140168135) > weight (0.25) 2023-12-02T03:03:33.026 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7457617317483854) > weight (0.25) 2023-12-02T03:03:33.026 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4468540559986016) > weight (0.25) 2023-12-02T03:03:33.026 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-12-02T03:03:33.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-12-02T03:03:33.026 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-02T03:03:33.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi057.stderr:2023-12-02T03:03:33.025+0000 7f4a10410640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 97863) UID: 1000 2023-12-02T03:03:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 600062976 bytes 2023-12-02T03:03:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 687466496 bytes 2023-12-02T03:03:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 748038144 bytes 2023-12-02T03:03:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 775135232 bytes 2023-12-02T03:03:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 819990528 bytes 2023-12-02T03:03:33.872 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 881064960 bytes 2023-12-02T03:03:33.981 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 895772672 bytes 2023-12-02T03:03:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 912029696 bytes 2023-12-02T03:03:34.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi057.stderr:2023-12-02T03:03:34.178+0000 7fa849031640 -1 rbd::mirror::ImageReplayer: 0x5573eb472500 [3/1f4ea1ff-7710-42b2-9726-194a70ed8f1e] register_admin_socket_hook: error registering admin socket commands 2023-12-02T03:03:34.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi057.stderr:2023-12-02T03:03:34.178+0000 7fa848830640 -1 rbd::mirror::ImageReplayer: 0x5573eb472500 [3/1f4ea1ff-7710-42b2-9726-194a70ed8f1e] register_admin_socket_hook: error registering admin socket commands 2023-12-02T03:03:34.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr:2023-12-02T03:03:34.190+0000 7f385a870640 -1 rbd::mirror::ImageReplayer: 0x56183025aa00 [3/1d600ef5-05fa-4ec8-bb36-7d891860097c] register_admin_socket_hook: error registering admin socket commands 2023-12-02T03:03:34.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr:2023-12-02T03:03:34.190+0000 7f385a06f640 -1 rbd::mirror::ImageReplayer: 0x56183025aa00 [3/1d600ef5-05fa-4ec8-bb36-7d891860097c] register_admin_socket_hook: error registering admin socket commands 2023-12-02T03:03:34.204 INFO:tasks.ceph.cluster2.osd.2.smithi057.stderr:2023-12-02T03:03:34.201+0000 7f34767b9640 -1 reset not still connected to 0x5622506a8750 2023-12-02T03:03:34.205 INFO:tasks.ceph.cluster2.osd.2.smithi057.stderr:2023-12-02T03:03:34.202+0000 7f34767b9640 -1 reset not still connected to 0x56225a3a8000 2023-12-02T03:03:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 925437952 bytes 2023-12-02T03:03:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 973652992 bytes 2023-12-02T03:03:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1029892096 bytes 2023-12-02T03:03:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1050712064 bytes 2023-12-02T03:03:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1066930176 bytes 2023-12-02T03:03:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1114246144 bytes 2023-12-02T03:03:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1177857024 bytes 2023-12-02T03:03:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1240417280 bytes 2023-12-02T03:03:35.366 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1277483008 bytes 2023-12-02T03:03:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1278117888 bytes 2023-12-02T03:03:35.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1306313728 bytes 2023-12-02T03:03:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1306932224 bytes 2023-12-02T03:03:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1348229120 bytes 2023-12-02T03:03:35.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1427729408 bytes 2023-12-02T03:03:36.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1439406080 bytes 2023-12-02T03:03:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1463906304 bytes 2023-12-02T03:03:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1545307136 bytes 2023-12-02T03:03:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1591653376 bytes 2023-12-02T03:03:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1657550848 bytes 2023-12-02T03:03:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1670322176 bytes 2023-12-02T03:03:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1719665664 bytes 2023-12-02T03:03:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1755035648 bytes 2023-12-02T03:03:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 1809101824 bytes 2023-12-02T03:03:37.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 1861659648 bytes 2023-12-02T03:03:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 1882217472 bytes 2023-12-02T03:03:37.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 1910094848 bytes 2023-12-02T03:03:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 1916005376 bytes 2023-12-02T03:03:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 1979980800 bytes 2023-12-02T03:03:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2073634816 bytes 2023-12-02T03:03:38.173 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2107344896 bytes 2023-12-02T03:03:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2134578176 bytes 2023-12-02T03:03:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2191418368 bytes 2023-12-02T03:03:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2238809088 bytes 2023-12-02T03:03:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2264961024 bytes 2023-12-02T03:03:38.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2293508096 bytes 2023-12-02T03:03:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2298092544 bytes 2023-12-02T03:03:39.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2314469376 bytes 2023-12-02T03:03:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2377370624 bytes 2023-12-02T03:03:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2454650880 bytes 2023-12-02T03:03:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2467815424 bytes 2023-12-02T03:03:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2532167680 bytes 2023-12-02T03:03:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:03:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:03:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:03:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:03:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:03:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_22 2023-12-02T03:03:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2023-12-02T03:03:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:03:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_22 2023-12-02T03:03:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:03:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:03:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_22 128M 2023-12-02T03:03:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:03:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_22 2023-12-02T03:03:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:03:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:03:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:03:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_22 2023-12-02T03:03:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:03:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:03:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:03:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2023-12-02T03:03:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:03:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_22 2023-12-02T03:03:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:03:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2023-12-02T03:03:40.027 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-12-02T03:03:40.027 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T03:03:40.027 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-12-02T03:03:40.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-02T03:03:40.028 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-02T03:03:40.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-02T03:03:40.030 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2023-12-02T03:03:40.530 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:03:40.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2023-12-02T03:03:40.774 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_22': 2023-12-02T03:03:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:03:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:03:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:03:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 137e47d4320d 2023-12-02T03:03:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.137e47d4320d 2023-12-02T03:03:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:03:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:03:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:03:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:03:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:03:39 2023 2023-12-02T03:03:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:03:39 2023 2023-12-02T03:03:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:03:39 2023 2023-12-02T03:03:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:03:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:03:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 8ef4933a-3981-486a-8a90-10311d67bf00 2023-12-02T03:03:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:03:40.778 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:03:40.778 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_22 2023-12-02T03:03:40.778 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:03:40.778 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_22 2023-12-02T03:03:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:03:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=11 2023-12-02T03:03:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:03:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:03:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 8349696 bytes 2023-12-02T03:03:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 35818496 bytes 2023-12-02T03:03:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 90326016 bytes 2023-12-02T03:03:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 120287232 bytes 2023-12-02T03:03:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 218298368 bytes 2023-12-02T03:03:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 237330432 bytes 2023-12-02T03:03:41.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 255572992 bytes 2023-12-02T03:03:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 304843776 bytes 2023-12-02T03:03:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 345043968 bytes 2023-12-02T03:03:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 437649408 bytes 2023-12-02T03:03:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 476391424 bytes 2023-12-02T03:03:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 501233664 bytes 2023-12-02T03:03:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 540920832 bytes 2023-12-02T03:03:42.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 609824768 bytes 2023-12-02T03:03:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 642055168 bytes 2023-12-02T03:03:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 754363392 bytes 2023-12-02T03:03:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 794484736 bytes 2023-12-02T03:03:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 818027520 bytes 2023-12-02T03:03:43.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 886093824 bytes 2023-12-02T03:03:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 901839872 bytes 2023-12-02T03:03:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 911854592 bytes 2023-12-02T03:03:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 938011648 bytes 2023-12-02T03:03:44.060 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 1059773440 bytes 2023-12-02T03:03:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1122351104 bytes 2023-12-02T03:03:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1157009408 bytes 2023-12-02T03:03:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1187786752 bytes 2023-12-02T03:03:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1252570112 bytes 2023-12-02T03:03:44.811 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1317627904 bytes 2023-12-02T03:03:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1358687232 bytes 2023-12-02T03:03:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1387184128 bytes 2023-12-02T03:03:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1400206336 bytes 2023-12-02T03:03:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1435741184 bytes 2023-12-02T03:03:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1469963264 bytes 2023-12-02T03:03:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1520476160 bytes 2023-12-02T03:03:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1560095744 bytes 2023-12-02T03:03:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1674190848 bytes 2023-12-02T03:03:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1735271424 bytes 2023-12-02T03:03:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1763041280 bytes 2023-12-02T03:03:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1782522880 bytes 2023-12-02T03:03:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1870129152 bytes 2023-12-02T03:03:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1930155008 bytes 2023-12-02T03:03:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1979762688 bytes 2023-12-02T03:03:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 2012864512 bytes 2023-12-02T03:03:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 2048764928 bytes 2023-12-02T03:03:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 2077953024 bytes 2023-12-02T03:03:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 2097148928 bytes 2023-12-02T03:03:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 2210319360 bytes 2023-12-02T03:03:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2235398144 bytes 2023-12-02T03:03:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2292043776 bytes 2023-12-02T03:03:47.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2332098560 bytes 2023-12-02T03:03:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2340244480 bytes 2023-12-02T03:03:48.242 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2405417984 bytes 2023-12-02T03:03:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2471041024 bytes 2023-12-02T03:03:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2501933056 bytes 2023-12-02T03:03:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2591924224 bytes 2023-12-02T03:03:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2635223040 bytes 2023-12-02T03:03:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2659491840 bytes 2023-12-02T03:03:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2714849280 bytes 2023-12-02T03:03:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2772116480 bytes 2023-12-02T03:03:49.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2800908288 bytes 2023-12-02T03:03:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2846978048 bytes 2023-12-02T03:03:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2869786624 bytes 2023-12-02T03:03:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2902295552 bytes 2023-12-02T03:03:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2968463360 bytes 2023-12-02T03:03:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2986145792 bytes 2023-12-02T03:03:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 3041438720 bytes 2023-12-02T03:03:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 3058193408 bytes 2023-12-02T03:03:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 3082808320 bytes 2023-12-02T03:03:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 3085806592 bytes 2023-12-02T03:03:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 3107418112 bytes 2023-12-02T03:03:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 3141716992 bytes 2023-12-02T03:03:51.102 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 3176030208 bytes 2023-12-02T03:03:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 3214754816 bytes 2023-12-02T03:03:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3258882048 bytes 2023-12-02T03:03:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3270209536 bytes 2023-12-02T03:03:51.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3338413056 bytes 2023-12-02T03:03:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:03:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:03:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:03:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:03:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:03:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_23 2023-12-02T03:03:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2023-12-02T03:03:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:03:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_23 2023-12-02T03:03:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:03:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:03:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_23 128M 2023-12-02T03:03:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:03:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_23 2023-12-02T03:03:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:03:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:03:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:03:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_23 2023-12-02T03:03:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:03:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:03:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:03:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2023-12-02T03:03:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:03:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_23 2023-12-02T03:03:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:03:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2023-12-02T03:03:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:03:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2023-12-02T03:03:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_23': 2023-12-02T03:03:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:03:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:03:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:03:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 1390f72c2e51 2023-12-02T03:03:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.1390f72c2e51 2023-12-02T03:03:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:03:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:03:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:03:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:03:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:03:51 2023 2023-12-02T03:03:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:03:51 2023 2023-12-02T03:03:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:03:51 2023 2023-12-02T03:03:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:03:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:03:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 8696705f-855b-402d-afd6-1c4d55e06c91 2023-12-02T03:03:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:03:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:03:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_23 2023-12-02T03:03:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:03:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_23 2023-12-02T03:03:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:03:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=7 2023-12-02T03:03:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:03:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:03:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 57028608 bytes 2023-12-02T03:03:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 98568192 bytes 2023-12-02T03:03:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 138835968 bytes 2023-12-02T03:03:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 150463488 bytes 2023-12-02T03:03:53.327 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 177304576 bytes 2023-12-02T03:03:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 243704832 bytes 2023-12-02T03:03:53.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 290357248 bytes 2023-12-02T03:03:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 342018048 bytes 2023-12-02T03:03:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 389433344 bytes 2023-12-02T03:03:54.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 411215872 bytes 2023-12-02T03:03:54.199 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 459561984 bytes 2023-12-02T03:03:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 506413056 bytes 2023-12-02T03:03:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 548676608 bytes 2023-12-02T03:03:54.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 593015808 bytes 2023-12-02T03:03:54.783 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 659614720 bytes 2023-12-02T03:03:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 686476288 bytes 2023-12-02T03:03:55.058 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 743313408 bytes 2023-12-02T03:03:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 779969536 bytes 2023-12-02T03:03:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 814668800 bytes 2023-12-02T03:03:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 889987072 bytes 2023-12-02T03:03:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 925446144 bytes 2023-12-02T03:03:55.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 956837888 bytes 2023-12-02T03:03:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 1018537984 bytes 2023-12-02T03:03:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1041978368 bytes 2023-12-02T03:03:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1078846464 bytes 2023-12-02T03:03:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1122402304 bytes 2023-12-02T03:03:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1156519936 bytes 2023-12-02T03:03:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1204498432 bytes 2023-12-02T03:03:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1218277376 bytes 2023-12-02T03:03:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1230136320 bytes 2023-12-02T03:03:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1267072000 bytes 2023-12-02T03:03:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1321240576 bytes 2023-12-02T03:03:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1342795776 bytes 2023-12-02T03:03:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1370575872 bytes 2023-12-02T03:03:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1408832512 bytes 2023-12-02T03:03:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1472829440 bytes 2023-12-02T03:03:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1495398400 bytes 2023-12-02T03:03:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1533831168 bytes 2023-12-02T03:03:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1534436352 bytes 2023-12-02T03:03:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1547606016 bytes 2023-12-02T03:03:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1612123136 bytes 2023-12-02T03:03:58.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1640089600 bytes 2023-12-02T03:03:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1730135040 bytes 2023-12-02T03:03:58.962 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1749713920 bytes 2023-12-02T03:03:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1781021696 bytes 2023-12-02T03:03:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1818060800 bytes 2023-12-02T03:03:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1857027072 bytes 2023-12-02T03:03:59.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 1937039360 bytes 2023-12-02T03:03:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:03:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:03:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:03:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:03:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:03:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_24 2023-12-02T03:03:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2023-12-02T03:03:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:03:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_24 2023-12-02T03:03:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:03:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:03:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_24 128M 2023-12-02T03:03:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:03:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_24 2023-12-02T03:03:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:03:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:03:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:03:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:03:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_24 2023-12-02T03:03:59.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:03:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:03:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:03:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2023-12-02T03:03:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:03:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:03:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_24 2023-12-02T03:03:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:03:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2023-12-02T03:04:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:04:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2023-12-02T03:04:00.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_24': 2023-12-02T03:04:00.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:04:00.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:04:00.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:04:00.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 139c6081492f 2023-12-02T03:04:00.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.139c6081492f 2023-12-02T03:04:00.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:04:00.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:04:00.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:04:00.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:04:00.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:03:59 2023 2023-12-02T03:04:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:03:59 2023 2023-12-02T03:04:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:03:59 2023 2023-12-02T03:04:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:04:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:04:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 9b3405db-a7dc-43f6-8462-f7325ff11d39 2023-12-02T03:04:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:04:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:04:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_24 2023-12-02T03:04:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:04:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:04:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_24 2023-12-02T03:04:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:04:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=9 2023-12-02T03:04:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:04:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:04:00.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 27448320 bytes 2023-12-02T03:04:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 28053504 bytes 2023-12-02T03:04:00.979 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 79082496 bytes 2023-12-02T03:04:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 111888384 bytes 2023-12-02T03:04:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 130068480 bytes 2023-12-02T03:04:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 154481664 bytes 2023-12-02T03:04:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 169163776 bytes 2023-12-02T03:04:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 220801024 bytes 2023-12-02T03:04:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 238950400 bytes 2023-12-02T03:04:01.786 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 254622720 bytes 2023-12-02T03:04:01.889 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 269778944 bytes 2023-12-02T03:04:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 298277888 bytes 2023-12-02T03:04:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 330519552 bytes 2023-12-02T03:04:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 407328768 bytes 2023-12-02T03:04:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 467653632 bytes 2023-12-02T03:04:02.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 501587968 bytes 2023-12-02T03:04:02.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 586161152 bytes 2023-12-02T03:04:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 618490880 bytes 2023-12-02T03:04:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 635614208 bytes 2023-12-02T03:04:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 671902720 bytes 2023-12-02T03:04:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 730462208 bytes 2023-12-02T03:04:03.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 731083776 bytes 2023-12-02T03:04:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 751858688 bytes 2023-12-02T03:04:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 800373760 bytes 2023-12-02T03:04:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 863501312 bytes 2023-12-02T03:04:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 897671168 bytes 2023-12-02T03:04:04.281 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 918288384 bytes 2023-12-02T03:04:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 963320832 bytes 2023-12-02T03:04:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1064300544 bytes 2023-12-02T03:04:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1119559680 bytes 2023-12-02T03:04:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1156796416 bytes 2023-12-02T03:04:05.081 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1197740032 bytes 2023-12-02T03:04:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1227042816 bytes 2023-12-02T03:04:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1281657856 bytes 2023-12-02T03:04:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1380855808 bytes 2023-12-02T03:04:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1381468160 bytes 2023-12-02T03:04:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1400672256 bytes 2023-12-02T03:04:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1442927616 bytes 2023-12-02T03:04:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1469819904 bytes 2023-12-02T03:04:06.153 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1479379968 bytes 2023-12-02T03:04:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1534455808 bytes 2023-12-02T03:04:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1555189760 bytes 2023-12-02T03:04:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1629498368 bytes 2023-12-02T03:04:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1703422976 bytes 2023-12-02T03:04:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1709412352 bytes 2023-12-02T03:04:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1786852352 bytes 2023-12-02T03:04:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1791313920 bytes 2023-12-02T03:04:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 1791941632 bytes 2023-12-02T03:04:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 1892078592 bytes 2023-12-02T03:04:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 1951549440 bytes 2023-12-02T03:04:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 1994587136 bytes 2023-12-02T03:04:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2026217472 bytes 2023-12-02T03:04:08.130 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2069956608 bytes 2023-12-02T03:04:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2115018752 bytes 2023-12-02T03:04:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2175389696 bytes 2023-12-02T03:04:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2194225152 bytes 2023-12-02T03:04:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2255370240 bytes 2023-12-02T03:04:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2327614464 bytes 2023-12-02T03:04:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2361903104 bytes 2023-12-02T03:04:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2431590400 bytes 2023-12-02T03:04:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2468274176 bytes 2023-12-02T03:04:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2517193728 bytes 2023-12-02T03:04:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2553683968 bytes 2023-12-02T03:04:09.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:04:09.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:04:09.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:04:09.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:04:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:04:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_25 2023-12-02T03:04:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2023-12-02T03:04:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:04:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:04:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_25 2023-12-02T03:04:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:04:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:04:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_25 128M 2023-12-02T03:04:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:04:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:04:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_25 2023-12-02T03:04:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:04:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:04:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:04:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_25 2023-12-02T03:04:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:04:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:04:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:04:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:04:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2023-12-02T03:04:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:04:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:04:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_25 2023-12-02T03:04:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:04:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2023-12-02T03:04:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:04:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2023-12-02T03:04:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_25': 2023-12-02T03:04:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:04:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:04:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:04:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 13a82940a991 2023-12-02T03:04:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.13a82940a991 2023-12-02T03:04:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:04:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:04:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:04:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:04:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:04:09 2023 2023-12-02T03:04:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:04:09 2023 2023-12-02T03:04:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:04:09 2023 2023-12-02T03:04:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:04:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:04:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 26af4e3d-22a4-46d1-af96-3f3836056461 2023-12-02T03:04:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:04:10.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:04:10.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_25 2023-12-02T03:04:10.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:04:10.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:04:10.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_25 2023-12-02T03:04:10.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:04:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=10 2023-12-02T03:04:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:04:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:04:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 56078336 bytes 2023-12-02T03:04:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 69264384 bytes 2023-12-02T03:04:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 113871872 bytes 2023-12-02T03:04:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 128107520 bytes 2023-12-02T03:04:11.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 215234560 bytes 2023-12-02T03:04:11.418 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 248904704 bytes 2023-12-02T03:04:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 297051136 bytes 2023-12-02T03:04:11.760 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr:2023-12-02T03:04:11.758+0000 7f60990ba640 -1 reset not still connected to 0x5611d0bd5110 2023-12-02T03:04:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 381925376 bytes 2023-12-02T03:04:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 422737920 bytes 2023-12-02T03:04:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 423361536 bytes 2023-12-02T03:04:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 477241344 bytes 2023-12-02T03:04:12.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 585062400 bytes 2023-12-02T03:04:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 640949248 bytes 2023-12-02T03:04:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 668300288 bytes 2023-12-02T03:04:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 687010816 bytes 2023-12-02T03:04:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 707382272 bytes 2023-12-02T03:04:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 731250688 bytes 2023-12-02T03:04:13.149 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 797732864 bytes 2023-12-02T03:04:13.296 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 841966592 bytes 2023-12-02T03:04:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 877095936 bytes 2023-12-02T03:04:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 913458176 bytes 2023-12-02T03:04:13.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 952544256 bytes 2023-12-02T03:04:13.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 963630080 bytes 2023-12-02T03:04:13.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 970660864 bytes 2023-12-02T03:04:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1032412160 bytes 2023-12-02T03:04:14.184 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1077501952 bytes 2023-12-02T03:04:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1101086720 bytes 2023-12-02T03:04:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1179566080 bytes 2023-12-02T03:04:14.656 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1247087616 bytes 2023-12-02T03:04:14.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1288834048 bytes 2023-12-02T03:04:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1372446720 bytes 2023-12-02T03:04:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1429100544 bytes 2023-12-02T03:04:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1483715584 bytes 2023-12-02T03:04:15.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1531209728 bytes 2023-12-02T03:04:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1606636544 bytes 2023-12-02T03:04:15.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1641606144 bytes 2023-12-02T03:04:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1650733056 bytes 2023-12-02T03:04:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1732421632 bytes 2023-12-02T03:04:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1771004928 bytes 2023-12-02T03:04:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1794318336 bytes 2023-12-02T03:04:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1818773504 bytes 2023-12-02T03:04:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1852943360 bytes 2023-12-02T03:04:16.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1889197056 bytes 2023-12-02T03:04:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1945133056 bytes 2023-12-02T03:04:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1988373504 bytes 2023-12-02T03:04:17.121 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 2041880576 bytes 2023-12-02T03:04:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 2103027712 bytes 2023-12-02T03:04:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2163819520 bytes 2023-12-02T03:04:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2187115520 bytes 2023-12-02T03:04:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2204832768 bytes 2023-12-02T03:04:17.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2258519040 bytes 2023-12-02T03:04:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2280591360 bytes 2023-12-02T03:04:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2340752384 bytes 2023-12-02T03:04:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2361186304 bytes 2023-12-02T03:04:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2388015104 bytes 2023-12-02T03:04:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2436214784 bytes 2023-12-02T03:04:18.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2452869120 bytes 2023-12-02T03:04:18.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2462374912 bytes 2023-12-02T03:04:18.985 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2520430592 bytes 2023-12-02T03:04:19.185 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2615030784 bytes 2023-12-02T03:04:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2679567360 bytes 2023-12-02T03:04:19.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2722396160 bytes 2023-12-02T03:04:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2762364928 bytes 2023-12-02T03:04:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2814619648 bytes 2023-12-02T03:04:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2840450048 bytes 2023-12-02T03:04:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2873573376 bytes 2023-12-02T03:04:20.124 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2874178560 bytes 2023-12-02T03:04:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2942620672 bytes 2023-12-02T03:04:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 3027566592 bytes 2023-12-02T03:04:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:04:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:04:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:04:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:04:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:04:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_26 2023-12-02T03:04:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2023-12-02T03:04:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:04:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:04:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_26 2023-12-02T03:04:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:04:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:04:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_26 128M 2023-12-02T03:04:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:04:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:04:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_26 2023-12-02T03:04:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:04:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:04:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:04:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_26 2023-12-02T03:04:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:04:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:04:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:04:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:04:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2023-12-02T03:04:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:04:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:04:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_26 2023-12-02T03:04:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:04:20.651 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2023-12-02T03:04:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:04:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2023-12-02T03:04:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_26': 2023-12-02T03:04:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:04:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:04:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:04:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 13b4e84bee27 2023-12-02T03:04:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.13b4e84bee27 2023-12-02T03:04:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:04:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:04:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:04:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:04:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:04:20 2023 2023-12-02T03:04:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:04:20 2023 2023-12-02T03:04:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:04:20 2023 2023-12-02T03:04:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:04:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:04:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 7dc110f7-a667-4d4c-9e52-4c6fe497aadb 2023-12-02T03:04:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:04:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:04:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_26 2023-12-02T03:04:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:04:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:04:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_26 2023-12-02T03:04:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:04:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=8 2023-12-02T03:04:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:04:21.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:04:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 39988224 bytes 2023-12-02T03:04:21.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 74398720 bytes 2023-12-02T03:04:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 99389440 bytes 2023-12-02T03:04:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 136285184 bytes 2023-12-02T03:04:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 208175104 bytes 2023-12-02T03:04:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 245993472 bytes 2023-12-02T03:04:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 272154624 bytes 2023-12-02T03:04:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 272773120 bytes 2023-12-02T03:04:22.798 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 295359488 bytes 2023-12-02T03:04:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 341818368 bytes 2023-12-02T03:04:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 373073920 bytes 2023-12-02T03:04:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 381293568 bytes 2023-12-02T03:04:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 441486336 bytes 2023-12-02T03:04:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 502458368 bytes 2023-12-02T03:04:23.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 585085952 bytes 2023-12-02T03:04:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 645741568 bytes 2023-12-02T03:04:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 691403776 bytes 2023-12-02T03:04:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 716597248 bytes 2023-12-02T03:04:24.333 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 833252352 bytes 2023-12-02T03:04:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 849145856 bytes 2023-12-02T03:04:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 866731008 bytes 2023-12-02T03:04:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 873185280 bytes 2023-12-02T03:04:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 946865152 bytes 2023-12-02T03:04:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 963904512 bytes 2023-12-02T03:04:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1004052480 bytes 2023-12-02T03:04:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1062801408 bytes 2023-12-02T03:04:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1082254336 bytes 2023-12-02T03:04:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1115154432 bytes 2023-12-02T03:04:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1199344640 bytes 2023-12-02T03:04:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1265829888 bytes 2023-12-02T03:04:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1277693952 bytes 2023-12-02T03:04:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1385891840 bytes 2023-12-02T03:04:26.323 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1429967872 bytes 2023-12-02T03:04:26.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1459945472 bytes 2023-12-02T03:04:26.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1507964928 bytes 2023-12-02T03:04:26.795 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1590973440 bytes 2023-12-02T03:04:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1611960320 bytes 2023-12-02T03:04:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1634386944 bytes 2023-12-02T03:04:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1658707968 bytes 2023-12-02T03:04:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1682191360 bytes 2023-12-02T03:04:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1710528512 bytes 2023-12-02T03:04:27.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1723365376 bytes 2023-12-02T03:04:27.692 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1799869440 bytes 2023-12-02T03:04:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1823750144 bytes 2023-12-02T03:04:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1904189440 bytes 2023-12-02T03:04:28.115 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1941547008 bytes 2023-12-02T03:04:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 2008389632 bytes 2023-12-02T03:04:28.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2022094848 bytes 2023-12-02T03:04:28.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2028529664 bytes 2023-12-02T03:04:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2031033344 bytes 2023-12-02T03:04:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2106555392 bytes 2023-12-02T03:04:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2143485952 bytes 2023-12-02T03:04:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2163470336 bytes 2023-12-02T03:04:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2209778688 bytes 2023-12-02T03:04:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2248890368 bytes 2023-12-02T03:04:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2285008896 bytes 2023-12-02T03:04:29.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2290870272 bytes 2023-12-02T03:04:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:04:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:04:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:04:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:04:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:04:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_27 2023-12-02T03:04:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2023-12-02T03:04:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:04:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:04:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_27 2023-12-02T03:04:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:04:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:04:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_27 128M 2023-12-02T03:04:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:04:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:04:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_27 2023-12-02T03:04:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:04:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:04:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:04:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_27 2023-12-02T03:04:29.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:04:29.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:04:29.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:04:29.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:04:29.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2023-12-02T03:04:29.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:04:29.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:04:29.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_27 2023-12-02T03:04:29.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:04:29.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2023-12-02T03:04:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:04:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2023-12-02T03:04:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_27': 2023-12-02T03:04:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:04:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:04:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:04:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 13c0c2b20973 2023-12-02T03:04:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.13c0c2b20973 2023-12-02T03:04:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:04:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:04:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:04:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:04:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:04:29 2023 2023-12-02T03:04:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:04:29 2023 2023-12-02T03:04:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:04:29 2023 2023-12-02T03:04:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:04:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:04:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 453a5e01-39a4-415e-bf21-d59460038eff 2023-12-02T03:04:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:04:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:04:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_27 2023-12-02T03:04:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:04:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:04:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_27 2023-12-02T03:04:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:04:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=13 2023-12-02T03:04:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:04:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:04:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 32301056 bytes 2023-12-02T03:04:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 64589824 bytes 2023-12-02T03:04:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 162430976 bytes 2023-12-02T03:04:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 202173440 bytes 2023-12-02T03:04:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 229579776 bytes 2023-12-02T03:04:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 266770432 bytes 2023-12-02T03:04:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 324803584 bytes 2023-12-02T03:04:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 364014592 bytes 2023-12-02T03:04:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 479815680 bytes 2023-12-02T03:04:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 525449216 bytes 2023-12-02T03:04:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 589680640 bytes 2023-12-02T03:04:32.657 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 635316224 bytes 2023-12-02T03:04:32.780 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 662800384 bytes 2023-12-02T03:04:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 758610944 bytes 2023-12-02T03:04:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 811064320 bytes 2023-12-02T03:04:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 848406528 bytes 2023-12-02T03:04:33.346 INFO:tasks.ceph.cluster2.osd.2.smithi057.stderr:2023-12-02T03:04:33.343+0000 7f34767b9640 -1 reset not still connected to 0x562254f80f70 2023-12-02T03:04:33.346 INFO:tasks.ceph.cluster2.osd.2.smithi057.stderr:2023-12-02T03:04:33.343+0000 7f34767b9640 -1 reset not still connected to 0x562259f076c0 2023-12-02T03:04:33.346 INFO:tasks.ceph.cluster2.osd.2.smithi057.stderr:2023-12-02T03:04:33.343+0000 7f34767b9640 -1 reset not still connected to 0x56225bab9790 2023-12-02T03:04:33.346 INFO:tasks.ceph.cluster2.osd.2.smithi057.stderr:2023-12-02T03:04:33.343+0000 7f34767b9640 -1 reset not still connected to 0x562268e609c0 2023-12-02T03:04:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 884416512 bytes 2023-12-02T03:04:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 912686080 bytes 2023-12-02T03:04:33.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 927428608 bytes 2023-12-02T03:04:33.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 995518464 bytes 2023-12-02T03:04:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 996120576 bytes 2023-12-02T03:04:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 1042010112 bytes 2023-12-02T03:04:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 1097728000 bytes 2023-12-02T03:04:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1137624064 bytes 2023-12-02T03:04:34.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1170239488 bytes 2023-12-02T03:04:34.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1204125696 bytes 2023-12-02T03:04:34.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1234518016 bytes 2023-12-02T03:04:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1324551168 bytes 2023-12-02T03:04:35.050 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1360801792 bytes 2023-12-02T03:04:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1381025792 bytes 2023-12-02T03:04:35.339 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1460722688 bytes 2023-12-02T03:04:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1497853952 bytes 2023-12-02T03:04:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1504901120 bytes 2023-12-02T03:04:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1543583744 bytes 2023-12-02T03:04:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1592589312 bytes 2023-12-02T03:04:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1632057344 bytes 2023-12-02T03:04:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1642458112 bytes 2023-12-02T03:04:36.207 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1657462784 bytes 2023-12-02T03:04:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1669145600 bytes 2023-12-02T03:04:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1683737600 bytes 2023-12-02T03:04:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1718909952 bytes 2023-12-02T03:04:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1728007168 bytes 2023-12-02T03:04:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1781074944 bytes 2023-12-02T03:04:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1802475520 bytes 2023-12-02T03:04:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1867667456 bytes 2023-12-02T03:04:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1891829760 bytes 2023-12-02T03:04:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1930816512 bytes 2023-12-02T03:04:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 1954575360 bytes 2023-12-02T03:04:37.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2035645440 bytes 2023-12-02T03:04:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2068620288 bytes 2023-12-02T03:04:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2128728064 bytes 2023-12-02T03:04:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2177370112 bytes 2023-12-02T03:04:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2230949888 bytes 2023-12-02T03:04:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2276372480 bytes 2023-12-02T03:04:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2312986624 bytes 2023-12-02T03:04:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2316635136 bytes 2023-12-02T03:04:38.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2365755392 bytes 2023-12-02T03:04:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2387269632 bytes 2023-12-02T03:04:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2441720832 bytes 2023-12-02T03:04:39.236 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2522527744 bytes 2023-12-02T03:04:39.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2576777216 bytes 2023-12-02T03:04:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2641647616 bytes 2023-12-02T03:04:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2707383296 bytes 2023-12-02T03:04:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2747941888 bytes 2023-12-02T03:04:39.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2748571648 bytes 2023-12-02T03:04:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2826821632 bytes 2023-12-02T03:04:40.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2859387904 bytes 2023-12-02T03:04:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2899147776 bytes 2023-12-02T03:04:40.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 2902165504 bytes 2023-12-02T03:04:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 2973216768 bytes 2023-12-02T03:04:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 3025932288 bytes 2023-12-02T03:04:40.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 3069467648 bytes 2023-12-02T03:04:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 3081086976 bytes 2023-12-02T03:04:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3151159296 bytes 2023-12-02T03:04:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3195659264 bytes 2023-12-02T03:04:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3309332480 bytes 2023-12-02T03:04:41.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3310922752 bytes 2023-12-02T03:04:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3376953344 bytes 2023-12-02T03:04:41.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3403660288 bytes 2023-12-02T03:04:42.122 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3436782592 bytes 2023-12-02T03:04:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3470109696 bytes 2023-12-02T03:04:42.377 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8200 IOs, 3503949824 bytes 2023-12-02T03:04:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8300 IOs, 3527488512 bytes 2023-12-02T03:04:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8400 IOs, 3554642944 bytes 2023-12-02T03:04:42.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8500 IOs, 3641805824 bytes 2023-12-02T03:04:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8600 IOs, 3705521152 bytes 2023-12-02T03:04:43.138 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-02T03:04:43.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-02T03:04:43.139 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-02T03:04:43.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr:2023-12-02T03:04:43.137+0000 7f70e76e9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 101547) UID: 1000 2023-12-02T03:04:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8700 IOs, 3777501184 bytes 2023-12-02T03:04:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8800 IOs, 3803053056 bytes 2023-12-02T03:04:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8900 IOs, 3845563392 bytes 2023-12-02T03:04:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9000 IOs, 3903071232 bytes 2023-12-02T03:04:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:04:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:04:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:04:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:04:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:04:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_28 2023-12-02T03:04:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2023-12-02T03:04:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:04:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:04:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_28 2023-12-02T03:04:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:04:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:04:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_28 128M 2023-12-02T03:04:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:04:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:04:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_28 2023-12-02T03:04:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:04:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:04:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:04:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_28 2023-12-02T03:04:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:04:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:04:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:04:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:04:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2023-12-02T03:04:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:04:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:04:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_28 2023-12-02T03:04:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:04:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2023-12-02T03:04:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:04:43.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2023-12-02T03:04:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_28': 2023-12-02T03:04:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:04:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:04:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:04:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 13cc3cfcae71 2023-12-02T03:04:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.13cc3cfcae71 2023-12-02T03:04:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:04:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:04:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:04:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:04:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:04:43 2023 2023-12-02T03:04:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:04:43 2023 2023-12-02T03:04:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:04:43 2023 2023-12-02T03:04:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:04:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:04:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: e7466612-311c-416f-a7f9-bfa20e74a575 2023-12-02T03:04:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:04:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:04:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_28 2023-12-02T03:04:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:04:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:04:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_28 2023-12-02T03:04:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:04:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=10 2023-12-02T03:04:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:04:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:04:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 23133184 bytes 2023-12-02T03:04:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 107956224 bytes 2023-12-02T03:04:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 170155008 bytes 2023-12-02T03:04:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 219825152 bytes 2023-12-02T03:04:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 258749440 bytes 2023-12-02T03:04:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 277639168 bytes 2023-12-02T03:04:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 305262592 bytes 2023-12-02T03:04:45.183 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 356454400 bytes 2023-12-02T03:04:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 413496320 bytes 2023-12-02T03:04:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 458991616 bytes 2023-12-02T03:04:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 494222336 bytes 2023-12-02T03:04:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 494848000 bytes 2023-12-02T03:04:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 536642560 bytes 2023-12-02T03:04:46.075 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 633084928 bytes 2023-12-02T03:04:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 660756480 bytes 2023-12-02T03:04:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 723829760 bytes 2023-12-02T03:04:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 746274816 bytes 2023-12-02T03:04:46.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 757396480 bytes 2023-12-02T03:04:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 778365952 bytes 2023-12-02T03:04:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 799685632 bytes 2023-12-02T03:04:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 816732160 bytes 2023-12-02T03:04:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 817381376 bytes 2023-12-02T03:04:47.149 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 866093056 bytes 2023-12-02T03:04:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 906488832 bytes 2023-12-02T03:04:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 960949248 bytes 2023-12-02T03:04:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 984997888 bytes 2023-12-02T03:04:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1034664960 bytes 2023-12-02T03:04:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1169329152 bytes 2023-12-02T03:04:48.099 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1202084864 bytes 2023-12-02T03:04:48.139 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-02T03:04:48.140 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T03:04:48.140 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-02T03:04:48.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-02T03:04:48.140 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-02T03:04:48.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-02T03:04:48.143 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2023-12-02T03:04:48.215 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1219524608 bytes 2023-12-02T03:04:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1239309312 bytes 2023-12-02T03:04:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1269669888 bytes 2023-12-02T03:04:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1313582080 bytes 2023-12-02T03:04:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1342438400 bytes 2023-12-02T03:04:48.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1413094400 bytes 2023-12-02T03:04:49.063 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1458794496 bytes 2023-12-02T03:04:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1522732032 bytes 2023-12-02T03:04:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1567322112 bytes 2023-12-02T03:04:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1652695040 bytes 2023-12-02T03:04:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1713427456 bytes 2023-12-02T03:04:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1716444160 bytes 2023-12-02T03:04:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1745262592 bytes 2023-12-02T03:04:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1783038976 bytes 2023-12-02T03:04:50.213 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1848636416 bytes 2023-12-02T03:04:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1885577216 bytes 2023-12-02T03:04:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1948004352 bytes 2023-12-02T03:04:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 2012978176 bytes 2023-12-02T03:04:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2013605888 bytes 2023-12-02T03:04:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2052684800 bytes 2023-12-02T03:04:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2114426880 bytes 2023-12-02T03:04:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2173657088 bytes 2023-12-02T03:04:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2216672256 bytes 2023-12-02T03:04:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2254674944 bytes 2023-12-02T03:04:51.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2299100160 bytes 2023-12-02T03:04:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2317649920 bytes 2023-12-02T03:04:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2370637824 bytes 2023-12-02T03:04:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2445521920 bytes 2023-12-02T03:04:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2508409856 bytes 2023-12-02T03:04:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2555872256 bytes 2023-12-02T03:04:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2596543488 bytes 2023-12-02T03:04:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2619094016 bytes 2023-12-02T03:04:52.770 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2619698176 bytes 2023-12-02T03:04:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2742134784 bytes 2023-12-02T03:04:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2845236224 bytes 2023-12-02T03:04:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2883068928 bytes 2023-12-02T03:04:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2892348416 bytes 2023-12-02T03:04:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2923830272 bytes 2023-12-02T03:04:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 3001659392 bytes 2023-12-02T03:04:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:04:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:04:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:04:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:04:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:04:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_29 2023-12-02T03:04:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2023-12-02T03:04:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:04:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:04:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_29 2023-12-02T03:04:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:04:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:04:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_29 128M 2023-12-02T03:04:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:04:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:04:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_29 2023-12-02T03:04:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:04:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:04:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:04:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:04:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_29 2023-12-02T03:04:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:04:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:04:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:04:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:04:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2023-12-02T03:04:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:04:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:04:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_29 2023-12-02T03:04:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:04:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2023-12-02T03:04:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:04:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2023-12-02T03:04:54.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_29': 2023-12-02T03:04:54.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:04:54.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:04:54.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:04:54.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 13e06edabef 2023-12-02T03:04:54.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.13e06edabef 2023-12-02T03:04:54.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:04:54.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:04:54.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:04:54.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:04:54.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:04:53 2023 2023-12-02T03:04:54.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:04:53 2023 2023-12-02T03:04:54.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:04:53 2023 2023-12-02T03:04:54.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:04:54.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:04:54.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 3d00c370-35ca-4837-a707-0aa3e5b4c7f4 2023-12-02T03:04:54.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:04:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:04:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_29 2023-12-02T03:04:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:04:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:04:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_29 2023-12-02T03:04:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:04:54.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=12 2023-12-02T03:04:54.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:04:54.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:04:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 107580416 bytes 2023-12-02T03:04:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 117059584 bytes 2023-12-02T03:04:55.092 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 153016320 bytes 2023-12-02T03:04:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 155777024 bytes 2023-12-02T03:04:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 162394112 bytes 2023-12-02T03:04:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 188596224 bytes 2023-12-02T03:04:55.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 228241408 bytes 2023-12-02T03:04:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 281431040 bytes 2023-12-02T03:04:55.786 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 308241408 bytes 2023-12-02T03:04:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 334294016 bytes 2023-12-02T03:04:56.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 364965888 bytes 2023-12-02T03:04:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 385891328 bytes 2023-12-02T03:04:56.312 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 447060992 bytes 2023-12-02T03:04:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 506710016 bytes 2023-12-02T03:04:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 554854400 bytes 2023-12-02T03:04:56.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 607416320 bytes 2023-12-02T03:04:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 678243328 bytes 2023-12-02T03:04:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 747296768 bytes 2023-12-02T03:04:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 784601088 bytes 2023-12-02T03:04:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 802712576 bytes 2023-12-02T03:04:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 856374272 bytes 2023-12-02T03:04:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 895778816 bytes 2023-12-02T03:04:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 935721984 bytes 2023-12-02T03:04:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 936343552 bytes 2023-12-02T03:04:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 975882240 bytes 2023-12-02T03:04:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1047586816 bytes 2023-12-02T03:04:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1111827456 bytes 2023-12-02T03:04:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1134370816 bytes 2023-12-02T03:04:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1161132032 bytes 2023-12-02T03:04:58.800 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1244817408 bytes 2023-12-02T03:04:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1282734080 bytes 2023-12-02T03:04:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1286701056 bytes 2023-12-02T03:04:59.154 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1319478272 bytes 2023-12-02T03:04:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1327981568 bytes 2023-12-02T03:04:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1428571136 bytes 2023-12-02T03:04:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1486462976 bytes 2023-12-02T03:04:59.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1563073536 bytes 2023-12-02T03:04:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1563677696 bytes 2023-12-02T03:05:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1597622272 bytes 2023-12-02T03:05:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1643439104 bytes 2023-12-02T03:05:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1644061696 bytes 2023-12-02T03:05:00.418 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1682123776 bytes 2023-12-02T03:05:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1699079168 bytes 2023-12-02T03:05:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1704595456 bytes 2023-12-02T03:05:00.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1728976896 bytes 2023-12-02T03:05:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1747005440 bytes 2023-12-02T03:05:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1775046656 bytes 2023-12-02T03:05:01.112 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 1828028416 bytes 2023-12-02T03:05:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 1829872640 bytes 2023-12-02T03:05:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 1855242240 bytes 2023-12-02T03:05:01.440 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 1886483456 bytes 2023-12-02T03:05:01.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 1914820608 bytes 2023-12-02T03:05:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 1931547648 bytes 2023-12-02T03:05:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 1984493568 bytes 2023-12-02T03:05:02.058 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2088907776 bytes 2023-12-02T03:05:02.173 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2117429248 bytes 2023-12-02T03:05:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2165517312 bytes 2023-12-02T03:05:02.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2167098368 bytes 2023-12-02T03:05:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2259444736 bytes 2023-12-02T03:05:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2313102336 bytes 2023-12-02T03:05:02.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2413522944 bytes 2023-12-02T03:05:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2428729344 bytes 2023-12-02T03:05:03.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2441593856 bytes 2023-12-02T03:05:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2508909568 bytes 2023-12-02T03:05:03.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2509926400 bytes 2023-12-02T03:05:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2533996544 bytes 2023-12-02T03:05:03.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2570836992 bytes 2023-12-02T03:05:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2638941184 bytes 2023-12-02T03:05:04.035 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 2699639808 bytes 2023-12-02T03:05:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 2749049856 bytes 2023-12-02T03:05:04.308 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 2784361472 bytes 2023-12-02T03:05:04.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 2809324544 bytes 2023-12-02T03:05:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 2866336768 bytes 2023-12-02T03:05:04.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 2930597888 bytes 2023-12-02T03:05:04.940 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3002022912 bytes 2023-12-02T03:05:05.098 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3057433600 bytes 2023-12-02T03:05:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3079834624 bytes 2023-12-02T03:05:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3144561664 bytes 2023-12-02T03:05:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3167629312 bytes 2023-12-02T03:05:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3206520832 bytes 2023-12-02T03:05:05.780 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3251399680 bytes 2023-12-02T03:05:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8200 IOs, 3302600704 bytes 2023-12-02T03:05:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8300 IOs, 3335179264 bytes 2023-12-02T03:05:06.193 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8400 IOs, 3384506368 bytes 2023-12-02T03:05:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8500 IOs, 3412420608 bytes 2023-12-02T03:05:06.470 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8600 IOs, 3448967168 bytes 2023-12-02T03:05:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:05:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:05:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:05:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:05:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:05:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_30 2023-12-02T03:05:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2023-12-02T03:05:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:05:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:05:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_30 2023-12-02T03:05:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:05:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:05:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_30 128M 2023-12-02T03:05:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:05:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:05:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_30 2023-12-02T03:05:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:05:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:05:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:05:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_30 2023-12-02T03:05:06.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:05:06.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:05:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:05:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:05:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2023-12-02T03:05:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:05:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:05:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_30 2023-12-02T03:05:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:05:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2023-12-02T03:05:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:05:07.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2023-12-02T03:05:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_30': 2023-12-02T03:05:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:05:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:05:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:05:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 13e8b828cab1 2023-12-02T03:05:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.13e8b828cab1 2023-12-02T03:05:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:05:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:05:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:05:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:05:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:05:06 2023 2023-12-02T03:05:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:05:06 2023 2023-12-02T03:05:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:05:06 2023 2023-12-02T03:05:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:05:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:05:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 03c2b3e5-9969-4a06-b152-32b7163be7b1 2023-12-02T03:05:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:05:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:05:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_30 2023-12-02T03:05:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:05:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:05:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_30 2023-12-02T03:05:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:05:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=5 2023-12-02T03:05:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:05:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:05:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 26198016 bytes 2023-12-02T03:05:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 113189888 bytes 2023-12-02T03:05:08.110 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 122900480 bytes 2023-12-02T03:05:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 142560256 bytes 2023-12-02T03:05:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 171235328 bytes 2023-12-02T03:05:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 182459392 bytes 2023-12-02T03:05:08.612 INFO:tasks.ceph.cluster2.osd.2.smithi057.stderr:2023-12-02T03:05:08.608+0000 7f34767b9640 -1 reset not still connected to 0x56225d686f70 2023-12-02T03:05:08.612 INFO:tasks.ceph.cluster2.osd.2.smithi057.stderr:2023-12-02T03:05:08.609+0000 7f34767b9640 -1 reset not still connected to 0x56225f661ad0 2023-12-02T03:05:08.613 INFO:tasks.ceph.cluster2.osd.2.smithi057.stderr:2023-12-02T03:05:08.609+0000 7f34767b9640 -1 reset not still connected to 0x56226172eea0 2023-12-02T03:05:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 255303680 bytes 2023-12-02T03:05:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 278272000 bytes 2023-12-02T03:05:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 311026688 bytes 2023-12-02T03:05:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 356300800 bytes 2023-12-02T03:05:09.152 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 392935424 bytes 2023-12-02T03:05:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 492890112 bytes 2023-12-02T03:05:09.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 502177792 bytes 2023-12-02T03:05:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 621302784 bytes 2023-12-02T03:05:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 664375296 bytes 2023-12-02T03:05:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 717703168 bytes 2023-12-02T03:05:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 750917632 bytes 2023-12-02T03:05:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 802914304 bytes 2023-12-02T03:05:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 859999232 bytes 2023-12-02T03:05:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 875650048 bytes 2023-12-02T03:05:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 920867840 bytes 2023-12-02T03:05:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 984629248 bytes 2023-12-02T03:05:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 1041008640 bytes 2023-12-02T03:05:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1088843776 bytes 2023-12-02T03:05:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1097556992 bytes 2023-12-02T03:05:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1098192896 bytes 2023-12-02T03:05:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1139142656 bytes 2023-12-02T03:05:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1159228416 bytes 2023-12-02T03:05:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1192821760 bytes 2023-12-02T03:05:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1227865088 bytes 2023-12-02T03:05:12.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1241322496 bytes 2023-12-02T03:05:12.134 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1245388800 bytes 2023-12-02T03:05:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1287928832 bytes 2023-12-02T03:05:12.364 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1300847616 bytes 2023-12-02T03:05:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1326532608 bytes 2023-12-02T03:05:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:05:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:05:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:05:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:05:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:05:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_31 2023-12-02T03:05:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2023-12-02T03:05:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:05:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:05:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:12.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_31 2023-12-02T03:05:12.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:12.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:05:12.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:05:12.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:12.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_31 128M 2023-12-02T03:05:12.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:05:12.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:12.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:05:12.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:12.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_31 2023-12-02T03:05:12.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:12.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:05:12.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:05:12.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:05:12.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:12.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_31 2023-12-02T03:05:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:05:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:05:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:05:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:05:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2023-12-02T03:05:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:05:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:05:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_31 2023-12-02T03:05:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:05:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2023-12-02T03:05:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:05:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2023-12-02T03:05:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_31': 2023-12-02T03:05:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:05:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:05:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:05:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 13f189bc41e 2023-12-02T03:05:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.13f189bc41e 2023-12-02T03:05:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:05:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:05:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:05:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:05:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:05:12 2023 2023-12-02T03:05:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:05:12 2023 2023-12-02T03:05:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:05:12 2023 2023-12-02T03:05:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:05:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:05:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: ffc4290c-3b4c-45f6-a30f-c93f3cb26275 2023-12-02T03:05:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:05:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:05:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_31 2023-12-02T03:05:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:05:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:05:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_31 2023-12-02T03:05:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:05:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=14 2023-12-02T03:05:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:05:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:05:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 87154688 bytes 2023-12-02T03:05:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 131638272 bytes 2023-12-02T03:05:14.084 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 152048640 bytes 2023-12-02T03:05:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 238973952 bytes 2023-12-02T03:05:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 294152192 bytes 2023-12-02T03:05:14.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 319217664 bytes 2023-12-02T03:05:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 400542720 bytes 2023-12-02T03:05:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 424597504 bytes 2023-12-02T03:05:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 470099968 bytes 2023-12-02T03:05:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 645996544 bytes 2023-12-02T03:05:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 703190016 bytes 2023-12-02T03:05:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 773477376 bytes 2023-12-02T03:05:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 779157504 bytes 2023-12-02T03:05:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 804480000 bytes 2023-12-02T03:05:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 835911680 bytes 2023-12-02T03:05:16.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 903535616 bytes 2023-12-02T03:05:16.251 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 953617408 bytes 2023-12-02T03:05:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 998153216 bytes 2023-12-02T03:05:16.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 1052378112 bytes 2023-12-02T03:05:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 1069148160 bytes 2023-12-02T03:05:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 1100992512 bytes 2023-12-02T03:05:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 1135714304 bytes 2023-12-02T03:05:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 1166807040 bytes 2023-12-02T03:05:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1189150720 bytes 2023-12-02T03:05:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1285092352 bytes 2023-12-02T03:05:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1329647616 bytes 2023-12-02T03:05:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1330242560 bytes 2023-12-02T03:05:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1354808320 bytes 2023-12-02T03:05:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1363200000 bytes 2023-12-02T03:05:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1398084608 bytes 2023-12-02T03:05:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1416646656 bytes 2023-12-02T03:05:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1427712000 bytes 2023-12-02T03:05:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1462622208 bytes 2023-12-02T03:05:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1470471168 bytes 2023-12-02T03:05:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1476126720 bytes 2023-12-02T03:05:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1512127488 bytes 2023-12-02T03:05:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1535712256 bytes 2023-12-02T03:05:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1575247872 bytes 2023-12-02T03:05:19.026 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1583901696 bytes 2023-12-02T03:05:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1615695872 bytes 2023-12-02T03:05:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1710915584 bytes 2023-12-02T03:05:19.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1750112256 bytes 2023-12-02T03:05:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1795066880 bytes 2023-12-02T03:05:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1806863360 bytes 2023-12-02T03:05:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1857786880 bytes 2023-12-02T03:05:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1858392064 bytes 2023-12-02T03:05:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1913818112 bytes 2023-12-02T03:05:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 1951739904 bytes 2023-12-02T03:05:20.449 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2015394816 bytes 2023-12-02T03:05:20.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2058584064 bytes 2023-12-02T03:05:20.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2087681024 bytes 2023-12-02T03:05:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2134939648 bytes 2023-12-02T03:05:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2174613504 bytes 2023-12-02T03:05:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2247575552 bytes 2023-12-02T03:05:21.341 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2295791616 bytes 2023-12-02T03:05:21.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2318860288 bytes 2023-12-02T03:05:21.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2352993280 bytes 2023-12-02T03:05:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2399301632 bytes 2023-12-02T03:05:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2459763712 bytes 2023-12-02T03:05:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2494792704 bytes 2023-12-02T03:05:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2514488320 bytes 2023-12-02T03:05:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2568045568 bytes 2023-12-02T03:05:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2620027904 bytes 2023-12-02T03:05:22.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2631391232 bytes 2023-12-02T03:05:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2676453376 bytes 2023-12-02T03:05:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2727183360 bytes 2023-12-02T03:05:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2800406528 bytes 2023-12-02T03:05:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2835266560 bytes 2023-12-02T03:05:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 2882656256 bytes 2023-12-02T03:05:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 2907572224 bytes 2023-12-02T03:05:23.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 2938398720 bytes 2023-12-02T03:05:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 2969355264 bytes 2023-12-02T03:05:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 3079825408 bytes 2023-12-02T03:05:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3130585088 bytes 2023-12-02T03:05:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3207025664 bytes 2023-12-02T03:05:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3250413568 bytes 2023-12-02T03:05:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3267824640 bytes 2023-12-02T03:05:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3315629056 bytes 2023-12-02T03:05:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3374604288 bytes 2023-12-02T03:05:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3382472704 bytes 2023-12-02T03:05:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3406913536 bytes 2023-12-02T03:05:25.172 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8200 IOs, 3412293632 bytes 2023-12-02T03:05:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8300 IOs, 3428273152 bytes 2023-12-02T03:05:25.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8400 IOs, 3461394432 bytes 2023-12-02T03:05:25.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8500 IOs, 3564056576 bytes 2023-12-02T03:05:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8600 IOs, 3634992128 bytes 2023-12-02T03:05:25.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8700 IOs, 3665341440 bytes 2023-12-02T03:05:26.107 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8800 IOs, 3716745216 bytes 2023-12-02T03:05:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8900 IOs, 3750735872 bytes 2023-12-02T03:05:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9000 IOs, 3755094016 bytes 2023-12-02T03:05:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9100 IOs, 3796111360 bytes 2023-12-02T03:05:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9200 IOs, 3868303360 bytes 2023-12-02T03:05:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9300 IOs, 3938099200 bytes 2023-12-02T03:05:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9400 IOs, 3984880640 bytes 2023-12-02T03:05:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9500 IOs, 3987651584 bytes 2023-12-02T03:05:27.237 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9600 IOs, 4035382272 bytes 2023-12-02T03:05:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9700 IOs, 4074823680 bytes 2023-12-02T03:05:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9800 IOs, 4092083200 bytes 2023-12-02T03:05:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9900 IOs, 4106949632 bytes 2023-12-02T03:05:27.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:05:27.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:05:27.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:05:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:05:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:05:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_32 2023-12-02T03:05:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2023-12-02T03:05:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:05:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:05:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_32 2023-12-02T03:05:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:05:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:05:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_32 128M 2023-12-02T03:05:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:05:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:05:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_32 2023-12-02T03:05:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:05:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:05:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:05:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_32 2023-12-02T03:05:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:05:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:05:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:05:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:05:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2023-12-02T03:05:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:05:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:05:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_32 2023-12-02T03:05:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:05:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2023-12-02T03:05:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:05:28.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2023-12-02T03:05:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_32': 2023-12-02T03:05:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:05:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:05:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:05:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 13f91bf5f503 2023-12-02T03:05:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.13f91bf5f503 2023-12-02T03:05:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:05:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:05:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:05:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:05:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:05:27 2023 2023-12-02T03:05:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:05:27 2023 2023-12-02T03:05:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:05:27 2023 2023-12-02T03:05:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:05:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:05:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: d4ceac3d-c782-416e-a7e1-be445a43ae35 2023-12-02T03:05:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:05:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:05:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_32 2023-12-02T03:05:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:05:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:05:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_32 2023-12-02T03:05:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:05:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=10 2023-12-02T03:05:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:05:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:05:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 24282112 bytes 2023-12-02T03:05:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 26378240 bytes 2023-12-02T03:05:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 88001536 bytes 2023-12-02T03:05:29.170 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 151716864 bytes 2023-12-02T03:05:29.276 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 174135296 bytes 2023-12-02T03:05:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 244395008 bytes 2023-12-02T03:05:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 289820672 bytes 2023-12-02T03:05:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 321311744 bytes 2023-12-02T03:05:29.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 362238976 bytes 2023-12-02T03:05:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 412552192 bytes 2023-12-02T03:05:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 421969920 bytes 2023-12-02T03:05:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 451883008 bytes 2023-12-02T03:05:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 534497280 bytes 2023-12-02T03:05:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 597379072 bytes 2023-12-02T03:05:30.773 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 676109312 bytes 2023-12-02T03:05:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 714769408 bytes 2023-12-02T03:05:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 737034240 bytes 2023-12-02T03:05:31.130 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 762639360 bytes 2023-12-02T03:05:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 787942400 bytes 2023-12-02T03:05:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 826518528 bytes 2023-12-02T03:05:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 852002816 bytes 2023-12-02T03:05:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 918520832 bytes 2023-12-02T03:05:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 962630656 bytes 2023-12-02T03:05:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1004945408 bytes 2023-12-02T03:05:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1035762688 bytes 2023-12-02T03:05:32.283 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1138481152 bytes 2023-12-02T03:05:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1194332160 bytes 2023-12-02T03:05:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1312442368 bytes 2023-12-02T03:05:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1324106752 bytes 2023-12-02T03:05:32.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1402336256 bytes 2023-12-02T03:05:33.031 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8336136865377756) > weight (0.25) 2023-12-02T03:05:33.032 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.2515563905114817) > weight (0.25) 2023-12-02T03:05:33.032 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.2947331481240347) > weight (0.25) 2023-12-02T03:05:33.032 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6173067472859662) > weight (0.25) 2023-12-02T03:05:33.032 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2023-12-02T03:05:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1463973888 bytes 2023-12-02T03:05:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1524688896 bytes 2023-12-02T03:05:33.442 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1606099968 bytes 2023-12-02T03:05:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1626382336 bytes 2023-12-02T03:05:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1680849920 bytes 2023-12-02T03:05:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1756734464 bytes 2023-12-02T03:05:34.032 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1808662528 bytes 2023-12-02T03:05:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1838857216 bytes 2023-12-02T03:05:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1865031680 bytes 2023-12-02T03:05:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1880089600 bytes 2023-12-02T03:05:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1942035456 bytes 2023-12-02T03:05:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1942642688 bytes 2023-12-02T03:05:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1943262208 bytes 2023-12-02T03:05:34.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1948191744 bytes 2023-12-02T03:05:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1980090368 bytes 2023-12-02T03:05:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 2000471040 bytes 2023-12-02T03:05:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 2026517504 bytes 2023-12-02T03:05:35.361 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2116016128 bytes 2023-12-02T03:05:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2191953920 bytes 2023-12-02T03:05:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2247566336 bytes 2023-12-02T03:05:35.811 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2252186624 bytes 2023-12-02T03:05:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2260178944 bytes 2023-12-02T03:05:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2285548544 bytes 2023-12-02T03:05:36.112 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2300820480 bytes 2023-12-02T03:05:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2338638848 bytes 2023-12-02T03:05:36.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2381396992 bytes 2023-12-02T03:05:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2423879680 bytes 2023-12-02T03:05:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2439545856 bytes 2023-12-02T03:05:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2512163840 bytes 2023-12-02T03:05:36.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2548801536 bytes 2023-12-02T03:05:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2663508992 bytes 2023-12-02T03:05:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2692658176 bytes 2023-12-02T03:05:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2755573760 bytes 2023-12-02T03:05:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2789455872 bytes 2023-12-02T03:05:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2833756160 bytes 2023-12-02T03:05:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2924554240 bytes 2023-12-02T03:05:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2985344000 bytes 2023-12-02T03:05:38.226 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 3015038976 bytes 2023-12-02T03:05:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 3064883200 bytes 2023-12-02T03:05:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 3126233088 bytes 2023-12-02T03:05:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:05:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:05:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:05:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:05:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:05:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_33 2023-12-02T03:05:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2023-12-02T03:05:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:05:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:05:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_33 2023-12-02T03:05:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:05:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:05:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_33 128M 2023-12-02T03:05:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:05:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:05:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_33 2023-12-02T03:05:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:05:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:05:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:05:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_33 2023-12-02T03:05:38.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:05:38.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:05:38.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:05:38.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:05:38.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2023-12-02T03:05:38.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:05:38.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:05:38.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_33 2023-12-02T03:05:38.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:05:38.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2023-12-02T03:05:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:05:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2023-12-02T03:05:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_33': 2023-12-02T03:05:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:05:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:05:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:05:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 1404845f2441 2023-12-02T03:05:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.1404845f2441 2023-12-02T03:05:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:05:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:05:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:05:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:05:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:05:38 2023 2023-12-02T03:05:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:05:38 2023 2023-12-02T03:05:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:05:38 2023 2023-12-02T03:05:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:05:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:05:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 56d75e57-f434-4172-81f2-851edf5c5eb5 2023-12-02T03:05:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:05:39.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:05:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_33 2023-12-02T03:05:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:05:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:05:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_33 2023-12-02T03:05:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:05:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=13 2023-12-02T03:05:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:05:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:05:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 18683904 bytes 2023-12-02T03:05:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 57742336 bytes 2023-12-02T03:05:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 100615168 bytes 2023-12-02T03:05:40.209 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 158519296 bytes 2023-12-02T03:05:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 182558720 bytes 2023-12-02T03:05:40.449 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 208352256 bytes 2023-12-02T03:05:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 286563328 bytes 2023-12-02T03:05:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 287156224 bytes 2023-12-02T03:05:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 288800768 bytes 2023-12-02T03:05:40.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 336578560 bytes 2023-12-02T03:05:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 374941696 bytes 2023-12-02T03:05:41.328 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 488259584 bytes 2023-12-02T03:05:41.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 526569472 bytes 2023-12-02T03:05:41.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 570287104 bytes 2023-12-02T03:05:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 609879040 bytes 2023-12-02T03:05:41.982 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 671163392 bytes 2023-12-02T03:05:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 700585984 bytes 2023-12-02T03:05:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 727391232 bytes 2023-12-02T03:05:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 728006656 bytes 2023-12-02T03:05:42.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 791940096 bytes 2023-12-02T03:05:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 822778880 bytes 2023-12-02T03:05:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 847625216 bytes 2023-12-02T03:05:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 887937024 bytes 2023-12-02T03:05:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 944357376 bytes 2023-12-02T03:05:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 967126016 bytes 2023-12-02T03:05:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 973145088 bytes 2023-12-02T03:05:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1065230336 bytes 2023-12-02T03:05:43.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1087513600 bytes 2023-12-02T03:05:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1088129024 bytes 2023-12-02T03:05:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1136891904 bytes 2023-12-02T03:05:43.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1156552704 bytes 2023-12-02T03:05:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1206358016 bytes 2023-12-02T03:05:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1219811328 bytes 2023-12-02T03:05:44.240 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1220438016 bytes 2023-12-02T03:05:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1290020864 bytes 2023-12-02T03:05:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1376395264 bytes 2023-12-02T03:05:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1413138432 bytes 2023-12-02T03:05:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1459385344 bytes 2023-12-02T03:05:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1492907008 bytes 2023-12-02T03:05:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1520698368 bytes 2023-12-02T03:05:45.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1581497344 bytes 2023-12-02T03:05:45.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1630632960 bytes 2023-12-02T03:05:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1694115840 bytes 2023-12-02T03:05:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1722938368 bytes 2023-12-02T03:05:45.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1814852608 bytes 2023-12-02T03:05:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1860244480 bytes 2023-12-02T03:05:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1944834048 bytes 2023-12-02T03:05:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2016215040 bytes 2023-12-02T03:05:46.529 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2048571392 bytes 2023-12-02T03:05:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2055437312 bytes 2023-12-02T03:05:46.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2072978432 bytes 2023-12-02T03:05:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2104397824 bytes 2023-12-02T03:05:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2190116864 bytes 2023-12-02T03:05:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2261578752 bytes 2023-12-02T03:05:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2262813696 bytes 2023-12-02T03:05:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2347652096 bytes 2023-12-02T03:05:47.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2377206784 bytes 2023-12-02T03:05:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2452384768 bytes 2023-12-02T03:05:47.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2503734272 bytes 2023-12-02T03:05:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2533984256 bytes 2023-12-02T03:05:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2589045760 bytes 2023-12-02T03:05:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2606606336 bytes 2023-12-02T03:05:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2648544256 bytes 2023-12-02T03:05:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2693979136 bytes 2023-12-02T03:05:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2712786944 bytes 2023-12-02T03:05:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2757260288 bytes 2023-12-02T03:05:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2796798976 bytes 2023-12-02T03:05:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2873496576 bytes 2023-12-02T03:05:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 2903806976 bytes 2023-12-02T03:05:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 2927306752 bytes 2023-12-02T03:05:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 2954343424 bytes 2023-12-02T03:05:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 2996539392 bytes 2023-12-02T03:05:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 3042654208 bytes 2023-12-02T03:05:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3055631360 bytes 2023-12-02T03:05:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3121031168 bytes 2023-12-02T03:05:50.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3163420672 bytes 2023-12-02T03:05:50.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3263729664 bytes 2023-12-02T03:05:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3299652608 bytes 2023-12-02T03:05:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3332182016 bytes 2023-12-02T03:05:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3378222080 bytes 2023-12-02T03:05:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3383941120 bytes 2023-12-02T03:05:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8200 IOs, 3411952640 bytes 2023-12-02T03:05:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8300 IOs, 3435601920 bytes 2023-12-02T03:05:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8400 IOs, 3455913984 bytes 2023-12-02T03:05:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8500 IOs, 3482363904 bytes 2023-12-02T03:05:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8600 IOs, 3496849408 bytes 2023-12-02T03:05:51.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8700 IOs, 3517319168 bytes 2023-12-02T03:05:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8800 IOs, 3569641472 bytes 2023-12-02T03:05:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8900 IOs, 3573092352 bytes 2023-12-02T03:05:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9000 IOs, 3605754880 bytes 2023-12-02T03:05:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9100 IOs, 3664519168 bytes 2023-12-02T03:05:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9200 IOs, 3679330304 bytes 2023-12-02T03:05:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9300 IOs, 3724134400 bytes 2023-12-02T03:05:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9400 IOs, 3749663744 bytes 2023-12-02T03:05:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:05:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:05:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:05:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:05:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:05:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_34 2023-12-02T03:05:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2023-12-02T03:05:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:05:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:05:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_34 2023-12-02T03:05:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:05:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:05:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_34 128M 2023-12-02T03:05:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:05:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:05:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_34 2023-12-02T03:05:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:05:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:05:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:05:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:05:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_34 2023-12-02T03:05:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:05:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:05:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:05:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:05:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2023-12-02T03:05:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:05:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:05:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_34 2023-12-02T03:05:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:05:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2023-12-02T03:05:53.174 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:05:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2023-12-02T03:05:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_34': 2023-12-02T03:05:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:05:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:05:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:05:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 140d7d54286e 2023-12-02T03:05:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.140d7d54286e 2023-12-02T03:05:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:05:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:05:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:05:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:05:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:05:52 2023 2023-12-02T03:05:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:05:52 2023 2023-12-02T03:05:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:05:52 2023 2023-12-02T03:05:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:05:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:05:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: e2407e91-d52b-4a70-b3dd-c23362ba73c5 2023-12-02T03:05:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:05:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:05:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_34 2023-12-02T03:05:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:05:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:05:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_34 2023-12-02T03:05:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:05:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=13 2023-12-02T03:05:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:05:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:05:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 91507712 bytes 2023-12-02T03:05:53.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 134552576 bytes 2023-12-02T03:05:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 210607104 bytes 2023-12-02T03:05:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 220792832 bytes 2023-12-02T03:05:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 262693888 bytes 2023-12-02T03:05:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 297691136 bytes 2023-12-02T03:05:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 323003392 bytes 2023-12-02T03:05:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 357168128 bytes 2023-12-02T03:05:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 412417024 bytes 2023-12-02T03:05:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 438905856 bytes 2023-12-02T03:05:54.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 504212480 bytes 2023-12-02T03:05:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 545103872 bytes 2023-12-02T03:05:55.086 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 569631744 bytes 2023-12-02T03:05:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 638475264 bytes 2023-12-02T03:05:55.367 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 651914240 bytes 2023-12-02T03:05:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 710929408 bytes 2023-12-02T03:05:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 716218368 bytes 2023-12-02T03:05:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 811945984 bytes 2023-12-02T03:05:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 848306176 bytes 2023-12-02T03:05:56.125 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 922725376 bytes 2023-12-02T03:05:56.292 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 979791872 bytes 2023-12-02T03:05:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 1038824448 bytes 2023-12-02T03:05:56.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 1041764352 bytes 2023-12-02T03:05:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1095247872 bytes 2023-12-02T03:05:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1144543232 bytes 2023-12-02T03:05:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1175521280 bytes 2023-12-02T03:05:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1249104896 bytes 2023-12-02T03:05:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1332766720 bytes 2023-12-02T03:05:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1351677952 bytes 2023-12-02T03:05:57.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1441593344 bytes 2023-12-02T03:05:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1458552832 bytes 2023-12-02T03:05:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1481319424 bytes 2023-12-02T03:05:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1508139008 bytes 2023-12-02T03:05:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1564354560 bytes 2023-12-02T03:05:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1639977984 bytes 2023-12-02T03:05:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1681589248 bytes 2023-12-02T03:05:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1699672064 bytes 2023-12-02T03:05:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1739427840 bytes 2023-12-02T03:05:58.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1755256832 bytes 2023-12-02T03:05:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1778968576 bytes 2023-12-02T03:05:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1827266560 bytes 2023-12-02T03:05:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1917060096 bytes 2023-12-02T03:05:59.415 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1947732992 bytes 2023-12-02T03:05:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1997047808 bytes 2023-12-02T03:05:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 2020600832 bytes 2023-12-02T03:05:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 2047047680 bytes 2023-12-02T03:05:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 2083735552 bytes 2023-12-02T03:06:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2161309696 bytes 2023-12-02T03:06:00.227 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2176583680 bytes 2023-12-02T03:06:00.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2205253632 bytes 2023-12-02T03:06:00.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2235632640 bytes 2023-12-02T03:06:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2316079104 bytes 2023-12-02T03:06:00.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2378247168 bytes 2023-12-02T03:06:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2471919616 bytes 2023-12-02T03:06:01.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2497815552 bytes 2023-12-02T03:06:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2520184832 bytes 2023-12-02T03:06:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2618518528 bytes 2023-12-02T03:06:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2657081344 bytes 2023-12-02T03:06:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2718025728 bytes 2023-12-02T03:06:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2762997760 bytes 2023-12-02T03:06:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2781293568 bytes 2023-12-02T03:06:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2839625728 bytes 2023-12-02T03:06:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2857351168 bytes 2023-12-02T03:06:02.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2897010688 bytes 2023-12-02T03:06:02.535 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2918960128 bytes 2023-12-02T03:06:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2988765184 bytes 2023-12-02T03:06:02.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 3015256064 bytes 2023-12-02T03:06:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 3067237376 bytes 2023-12-02T03:06:03.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 3067825152 bytes 2023-12-02T03:06:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 3124386816 bytes 2023-12-02T03:06:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 3169855488 bytes 2023-12-02T03:06:03.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 3189285888 bytes 2023-12-02T03:06:03.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 3218249728 bytes 2023-12-02T03:06:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3236709376 bytes 2023-12-02T03:06:03.856 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3294615552 bytes 2023-12-02T03:06:03.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3332635648 bytes 2023-12-02T03:06:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3398383616 bytes 2023-12-02T03:06:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3447271424 bytes 2023-12-02T03:06:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3494816768 bytes 2023-12-02T03:06:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3532392448 bytes 2023-12-02T03:06:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3573352448 bytes 2023-12-02T03:06:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8200 IOs, 3604003840 bytes 2023-12-02T03:06:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8300 IOs, 3659288576 bytes 2023-12-02T03:06:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8400 IOs, 3697153024 bytes 2023-12-02T03:06:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8500 IOs, 3760261120 bytes 2023-12-02T03:06:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8600 IOs, 3832771584 bytes 2023-12-02T03:06:05.637 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8700 IOs, 3907127296 bytes 2023-12-02T03:06:05.810 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8800 IOs, 3975938048 bytes 2023-12-02T03:06:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8900 IOs, 4060913664 bytes 2023-12-02T03:06:06.134 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9000 IOs, 4092181504 bytes 2023-12-02T03:06:06.231 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:06:06.231 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:06:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:06:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:06:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:06:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_35 2023-12-02T03:06:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2023-12-02T03:06:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:06:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:06:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_35 2023-12-02T03:06:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:06:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:06:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_35 128M 2023-12-02T03:06:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:06:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:06:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_35 2023-12-02T03:06:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:06:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:06:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:06:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_35 2023-12-02T03:06:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:06:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:06:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:06:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:06:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2023-12-02T03:06:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:06:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:06:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_35 2023-12-02T03:06:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:06:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2023-12-02T03:06:06.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:06:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2023-12-02T03:06:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_35': 2023-12-02T03:06:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:06:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:06:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:06:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 1417870b26c7 2023-12-02T03:06:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.1417870b26c7 2023-12-02T03:06:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:06:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:06:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:06:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:06:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:06:06 2023 2023-12-02T03:06:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:06:06 2023 2023-12-02T03:06:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:06:06 2023 2023-12-02T03:06:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:06:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:06:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 9802897e-7ccb-4bb2-8ab0-cedc09b0044e 2023-12-02T03:06:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:06:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:06:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_35 2023-12-02T03:06:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:06:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:06:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_35 2023-12-02T03:06:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:06:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=10 2023-12-02T03:06:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:06:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:06:06.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 92680192 bytes 2023-12-02T03:06:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 101586944 bytes 2023-12-02T03:06:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 181688320 bytes 2023-12-02T03:06:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 209203200 bytes 2023-12-02T03:06:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 229587968 bytes 2023-12-02T03:06:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 265905152 bytes 2023-12-02T03:06:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 308424704 bytes 2023-12-02T03:06:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 329876480 bytes 2023-12-02T03:06:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 447840256 bytes 2023-12-02T03:06:08.153 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 479638528 bytes 2023-12-02T03:06:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 505903104 bytes 2023-12-02T03:06:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 534696960 bytes 2023-12-02T03:06:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 571452416 bytes 2023-12-02T03:06:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 630524928 bytes 2023-12-02T03:06:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 690977792 bytes 2023-12-02T03:06:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 741203968 bytes 2023-12-02T03:06:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 755864576 bytes 2023-12-02T03:06:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 777157632 bytes 2023-12-02T03:06:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 856623104 bytes 2023-12-02T03:06:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 882146304 bytes 2023-12-02T03:06:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 937405440 bytes 2023-12-02T03:06:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 938535936 bytes 2023-12-02T03:06:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 939128832 bytes 2023-12-02T03:06:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 942020608 bytes 2023-12-02T03:06:10.174 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1047566336 bytes 2023-12-02T03:06:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1097075712 bytes 2023-12-02T03:06:10.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1141778432 bytes 2023-12-02T03:06:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1224255488 bytes 2023-12-02T03:06:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1266417664 bytes 2023-12-02T03:06:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1313543168 bytes 2023-12-02T03:06:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1353862144 bytes 2023-12-02T03:06:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1397513216 bytes 2023-12-02T03:06:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1439078400 bytes 2023-12-02T03:06:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1440478208 bytes 2023-12-02T03:06:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1476366336 bytes 2023-12-02T03:06:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1507118080 bytes 2023-12-02T03:06:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1543511040 bytes 2023-12-02T03:06:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1594405888 bytes 2023-12-02T03:06:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1647761408 bytes 2023-12-02T03:06:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1666391040 bytes 2023-12-02T03:06:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1686495232 bytes 2023-12-02T03:06:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1719566336 bytes 2023-12-02T03:06:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1782230016 bytes 2023-12-02T03:06:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1826899968 bytes 2023-12-02T03:06:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1840096256 bytes 2023-12-02T03:06:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1856113664 bytes 2023-12-02T03:06:13.357 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1866942464 bytes 2023-12-02T03:06:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 1904941056 bytes 2023-12-02T03:06:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 1925420032 bytes 2023-12-02T03:06:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2022511616 bytes 2023-12-02T03:06:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2125290496 bytes 2023-12-02T03:06:14.190 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2181820416 bytes 2023-12-02T03:06:14.314 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2212296704 bytes 2023-12-02T03:06:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2250494976 bytes 2023-12-02T03:06:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2300326912 bytes 2023-12-02T03:06:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2344012800 bytes 2023-12-02T03:06:14.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2424163328 bytes 2023-12-02T03:06:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2469740544 bytes 2023-12-02T03:06:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2579574784 bytes 2023-12-02T03:06:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2618194944 bytes 2023-12-02T03:06:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2642193408 bytes 2023-12-02T03:06:15.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2670632960 bytes 2023-12-02T03:06:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2734448640 bytes 2023-12-02T03:06:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2775585792 bytes 2023-12-02T03:06:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2784618496 bytes 2023-12-02T03:06:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2867977216 bytes 2023-12-02T03:06:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2890675200 bytes 2023-12-02T03:06:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2891313152 bytes 2023-12-02T03:06:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 2948815872 bytes 2023-12-02T03:06:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:06:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:06:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:06:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:06:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:06:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_36 2023-12-02T03:06:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2023-12-02T03:06:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:06:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:06:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_36 2023-12-02T03:06:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:06:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:06:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_36 128M 2023-12-02T03:06:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:06:16.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:16.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:06:16.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:16.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_36 2023-12-02T03:06:16.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:16.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:06:16.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:06:16.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:06:16.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:16.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_36 2023-12-02T03:06:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:06:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:06:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:06:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:06:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2023-12-02T03:06:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:06:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:06:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_36 2023-12-02T03:06:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:06:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2023-12-02T03:06:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:06:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2023-12-02T03:06:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_36': 2023-12-02T03:06:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:06:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:06:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:06:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 14207e48659e 2023-12-02T03:06:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.14207e48659e 2023-12-02T03:06:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:06:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:06:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:06:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:06:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:06:16 2023 2023-12-02T03:06:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:06:16 2023 2023-12-02T03:06:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:06:16 2023 2023-12-02T03:06:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:06:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:06:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 5c374561-6563-424e-bb48-6c5014bde1fc 2023-12-02T03:06:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:06:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:06:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_36 2023-12-02T03:06:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:06:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:06:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_36 2023-12-02T03:06:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:06:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=13 2023-12-02T03:06:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:06:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:06:17.796 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 1454080 bytes 2023-12-02T03:06:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 31969280 bytes 2023-12-02T03:06:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 77629440 bytes 2023-12-02T03:06:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 131779584 bytes 2023-12-02T03:06:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 185382912 bytes 2023-12-02T03:06:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 246836224 bytes 2023-12-02T03:06:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 284824576 bytes 2023-12-02T03:06:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 348562432 bytes 2023-12-02T03:06:18.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 381677568 bytes 2023-12-02T03:06:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 436668416 bytes 2023-12-02T03:06:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 451373056 bytes 2023-12-02T03:06:19.362 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 492795904 bytes 2023-12-02T03:06:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 514366464 bytes 2023-12-02T03:06:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 560501760 bytes 2023-12-02T03:06:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 586370048 bytes 2023-12-02T03:06:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 620408832 bytes 2023-12-02T03:06:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 662088704 bytes 2023-12-02T03:06:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 705546240 bytes 2023-12-02T03:06:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 737177600 bytes 2023-12-02T03:06:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 757475328 bytes 2023-12-02T03:06:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 800994304 bytes 2023-12-02T03:06:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 801615872 bytes 2023-12-02T03:06:20.791 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 844929024 bytes 2023-12-02T03:06:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 890278912 bytes 2023-12-02T03:06:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 907589632 bytes 2023-12-02T03:06:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 921587712 bytes 2023-12-02T03:06:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 965429248 bytes 2023-12-02T03:06:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1006660608 bytes 2023-12-02T03:06:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1011263488 bytes 2023-12-02T03:06:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1049890816 bytes 2023-12-02T03:06:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1092030464 bytes 2023-12-02T03:06:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1138049024 bytes 2023-12-02T03:06:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1181473792 bytes 2023-12-02T03:06:22.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1250314240 bytes 2023-12-02T03:06:22.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1349331968 bytes 2023-12-02T03:06:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1355866112 bytes 2023-12-02T03:06:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1408612352 bytes 2023-12-02T03:06:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1417164800 bytes 2023-12-02T03:06:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1457934336 bytes 2023-12-02T03:06:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1522177024 bytes 2023-12-02T03:06:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1572443136 bytes 2023-12-02T03:06:23.357 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1580713984 bytes 2023-12-02T03:06:23.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1658697728 bytes 2023-12-02T03:06:23.656 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1695535104 bytes 2023-12-02T03:06:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1706385408 bytes 2023-12-02T03:06:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1772184576 bytes 2023-12-02T03:06:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1853211648 bytes 2023-12-02T03:06:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 1871549440 bytes 2023-12-02T03:06:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 1927916544 bytes 2023-12-02T03:06:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 1986238464 bytes 2023-12-02T03:06:24.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2019612672 bytes 2023-12-02T03:06:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2077171712 bytes 2023-12-02T03:06:25.113 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2102447104 bytes 2023-12-02T03:06:25.243 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2132965376 bytes 2023-12-02T03:06:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2173700096 bytes 2023-12-02T03:06:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2218564608 bytes 2023-12-02T03:06:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2290592768 bytes 2023-12-02T03:06:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2348076032 bytes 2023-12-02T03:06:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2361947136 bytes 2023-12-02T03:06:26.107 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2396817408 bytes 2023-12-02T03:06:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2427234304 bytes 2023-12-02T03:06:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2524867584 bytes 2023-12-02T03:06:26.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2620506112 bytes 2023-12-02T03:06:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2670996480 bytes 2023-12-02T03:06:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2712845312 bytes 2023-12-02T03:06:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2738143232 bytes 2023-12-02T03:06:27.200 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2780651520 bytes 2023-12-02T03:06:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2823958528 bytes 2023-12-02T03:06:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 2866853888 bytes 2023-12-02T03:06:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 2876585984 bytes 2023-12-02T03:06:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 2877999104 bytes 2023-12-02T03:06:27.798 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 2890342400 bytes 2023-12-02T03:06:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 2904006656 bytes 2023-12-02T03:06:28.086 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 2961809408 bytes 2023-12-02T03:06:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3005385728 bytes 2023-12-02T03:06:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3027323904 bytes 2023-12-02T03:06:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3074733056 bytes 2023-12-02T03:06:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3160214528 bytes 2023-12-02T03:06:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3198025728 bytes 2023-12-02T03:06:28.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3235408896 bytes 2023-12-02T03:06:29.200 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3345500160 bytes 2023-12-02T03:06:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8200 IOs, 3437024256 bytes 2023-12-02T03:06:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8300 IOs, 3451855872 bytes 2023-12-02T03:06:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8400 IOs, 3502478336 bytes 2023-12-02T03:06:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8500 IOs, 3562429440 bytes 2023-12-02T03:06:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8600 IOs, 3614112768 bytes 2023-12-02T03:06:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8700 IOs, 3630032896 bytes 2023-12-02T03:06:30.168 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8800 IOs, 3650605056 bytes 2023-12-02T03:06:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8900 IOs, 3690015744 bytes 2023-12-02T03:06:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9000 IOs, 3725191168 bytes 2023-12-02T03:06:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9100 IOs, 3730300928 bytes 2023-12-02T03:06:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:06:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:06:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:06:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:06:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:06:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_37 2023-12-02T03:06:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2023-12-02T03:06:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:06:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:06:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_37 2023-12-02T03:06:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:06:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:06:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_37 128M 2023-12-02T03:06:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:06:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:06:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_37 2023-12-02T03:06:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:06:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:06:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:06:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_37 2023-12-02T03:06:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:06:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:06:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:06:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:06:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2023-12-02T03:06:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:06:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:06:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_37 2023-12-02T03:06:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:06:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2023-12-02T03:06:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:06:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2023-12-02T03:06:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_37': 2023-12-02T03:06:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:06:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:06:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:06:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 1428be9daa94 2023-12-02T03:06:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.1428be9daa94 2023-12-02T03:06:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:06:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:06:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:06:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:06:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:06:30 2023 2023-12-02T03:06:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:06:30 2023 2023-12-02T03:06:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:06:30 2023 2023-12-02T03:06:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:06:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:06:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 9333a204-121f-424e-849b-e6aec78f4224 2023-12-02T03:06:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:06:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:06:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_37 2023-12-02T03:06:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:06:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:06:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_37 2023-12-02T03:06:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:06:31.680 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=13 2023-12-02T03:06:31.680 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:06:31.680 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:06:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 75748352 bytes 2023-12-02T03:06:32.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 128672768 bytes 2023-12-02T03:06:32.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 239083520 bytes 2023-12-02T03:06:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 278631424 bytes 2023-12-02T03:06:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 316832768 bytes 2023-12-02T03:06:32.709 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 392013824 bytes 2023-12-02T03:06:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 440551424 bytes 2023-12-02T03:06:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 490674176 bytes 2023-12-02T03:06:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 562918400 bytes 2023-12-02T03:06:33.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 563552256 bytes 2023-12-02T03:06:33.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 619885568 bytes 2023-12-02T03:06:33.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 659568640 bytes 2023-12-02T03:06:33.717 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 716537856 bytes 2023-12-02T03:06:33.872 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 763267072 bytes 2023-12-02T03:06:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 840390656 bytes 2023-12-02T03:06:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 861288448 bytes 2023-12-02T03:06:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 909838336 bytes 2023-12-02T03:06:34.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 927123456 bytes 2023-12-02T03:06:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 937857024 bytes 2023-12-02T03:06:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 972085248 bytes 2023-12-02T03:06:34.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 1005811712 bytes 2023-12-02T03:06:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 1041623040 bytes 2023-12-02T03:06:35.082 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 1110077440 bytes 2023-12-02T03:06:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1147515904 bytes 2023-12-02T03:06:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1182780416 bytes 2023-12-02T03:06:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1264724992 bytes 2023-12-02T03:06:35.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1297759232 bytes 2023-12-02T03:06:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1383377920 bytes 2023-12-02T03:06:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1414487040 bytes 2023-12-02T03:06:36.112 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1443491840 bytes 2023-12-02T03:06:36.202 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1461433344 bytes 2023-12-02T03:06:36.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1539658752 bytes 2023-12-02T03:06:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1553707008 bytes 2023-12-02T03:06:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1569902592 bytes 2023-12-02T03:06:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1585413120 bytes 2023-12-02T03:06:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1606178816 bytes 2023-12-02T03:06:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1620715520 bytes 2023-12-02T03:06:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1711566848 bytes 2023-12-02T03:06:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1729928192 bytes 2023-12-02T03:06:37.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1742902272 bytes 2023-12-02T03:06:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1760142336 bytes 2023-12-02T03:06:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1835913216 bytes 2023-12-02T03:06:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1874591744 bytes 2023-12-02T03:06:37.912 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1899692032 bytes 2023-12-02T03:06:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1917513728 bytes 2023-12-02T03:06:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1961358336 bytes 2023-12-02T03:06:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1961974784 bytes 2023-12-02T03:06:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2060035072 bytes 2023-12-02T03:06:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2110604288 bytes 2023-12-02T03:06:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2150909952 bytes 2023-12-02T03:06:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2173093888 bytes 2023-12-02T03:06:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2222878720 bytes 2023-12-02T03:06:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2254732288 bytes 2023-12-02T03:06:39.246 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2290865152 bytes 2023-12-02T03:06:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2355651584 bytes 2023-12-02T03:06:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2373218304 bytes 2023-12-02T03:06:39.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2373833728 bytes 2023-12-02T03:06:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2412558336 bytes 2023-12-02T03:06:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2434579456 bytes 2023-12-02T03:06:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2463661056 bytes 2023-12-02T03:06:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2567686144 bytes 2023-12-02T03:06:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2632346624 bytes 2023-12-02T03:06:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2656929792 bytes 2023-12-02T03:06:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2675555328 bytes 2023-12-02T03:06:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2699853824 bytes 2023-12-02T03:06:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2728248320 bytes 2023-12-02T03:06:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2728858624 bytes 2023-12-02T03:06:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2755129344 bytes 2023-12-02T03:06:41.144 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4637865037209039) > weight (0.25) 2023-12-02T03:06:41.144 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8081613524411034) > weight (0.25) 2023-12-02T03:06:41.145 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.29152768824361686) > weight (0.25) 2023-12-02T03:06:41.145 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8639109147035255) > weight (0.25) 2023-12-02T03:06:41.145 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2023-12-02T03:06:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 2829957120 bytes 2023-12-02T03:06:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 2902934528 bytes 2023-12-02T03:06:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 2929656832 bytes 2023-12-02T03:06:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 2963251200 bytes 2023-12-02T03:06:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 2982439936 bytes 2023-12-02T03:06:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3016953856 bytes 2023-12-02T03:06:42.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3057532928 bytes 2023-12-02T03:06:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3088340992 bytes 2023-12-02T03:06:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3094369280 bytes 2023-12-02T03:06:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3139443712 bytes 2023-12-02T03:06:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3155204096 bytes 2023-12-02T03:06:42.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3226485760 bytes 2023-12-02T03:06:42.860 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3263620096 bytes 2023-12-02T03:06:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8200 IOs, 3264212992 bytes 2023-12-02T03:06:43.142 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8300 IOs, 3357357056 bytes 2023-12-02T03:06:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8400 IOs, 3378072576 bytes 2023-12-02T03:06:43.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8500 IOs, 3397489664 bytes 2023-12-02T03:06:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8600 IOs, 3417353216 bytes 2023-12-02T03:06:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8700 IOs, 3452680192 bytes 2023-12-02T03:06:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8800 IOs, 3453287424 bytes 2023-12-02T03:06:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8900 IOs, 3477288960 bytes 2023-12-02T03:06:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9000 IOs, 3514196992 bytes 2023-12-02T03:06:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9100 IOs, 3578206208 bytes 2023-12-02T03:06:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9200 IOs, 3583092736 bytes 2023-12-02T03:06:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9300 IOs, 3669798912 bytes 2023-12-02T03:06:44.578 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9400 IOs, 3710768128 bytes 2023-12-02T03:06:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:06:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:06:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:06:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:06:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:06:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_38 2023-12-02T03:06:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2023-12-02T03:06:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:06:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:06:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_38 2023-12-02T03:06:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:06:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:06:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_38 128M 2023-12-02T03:06:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:06:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:06:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_38 2023-12-02T03:06:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:06:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:06:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:06:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:44.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_38 2023-12-02T03:06:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:06:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:06:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:06:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:06:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2023-12-02T03:06:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:06:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:06:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_38 2023-12-02T03:06:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:06:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2023-12-02T03:06:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:06:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2023-12-02T03:06:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_38': 2023-12-02T03:06:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:06:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:06:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:06:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 1430f3060df4 2023-12-02T03:06:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.1430f3060df4 2023-12-02T03:06:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:06:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:06:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:06:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:06:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:06:44 2023 2023-12-02T03:06:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:06:44 2023 2023-12-02T03:06:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:06:44 2023 2023-12-02T03:06:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:06:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:06:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 8a609a08-21d0-4193-a27d-f581f4516454 2023-12-02T03:06:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:06:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:06:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_38 2023-12-02T03:06:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:06:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:06:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_38 2023-12-02T03:06:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:06:45.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=10 2023-12-02T03:06:45.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:06:45.651 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:06:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 35808256 bytes 2023-12-02T03:06:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 101249024 bytes 2023-12-02T03:06:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 119649280 bytes 2023-12-02T03:06:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 138530816 bytes 2023-12-02T03:06:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 168232960 bytes 2023-12-02T03:06:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 232599552 bytes 2023-12-02T03:06:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 269427712 bytes 2023-12-02T03:06:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 324491264 bytes 2023-12-02T03:06:47.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 346300416 bytes 2023-12-02T03:06:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 381819904 bytes 2023-12-02T03:06:47.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 416609280 bytes 2023-12-02T03:06:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 443881472 bytes 2023-12-02T03:06:47.741 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 452198400 bytes 2023-12-02T03:06:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 488012800 bytes 2023-12-02T03:06:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 522910720 bytes 2023-12-02T03:06:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 569058304 bytes 2023-12-02T03:06:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 594789376 bytes 2023-12-02T03:06:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 726311936 bytes 2023-12-02T03:06:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 732767232 bytes 2023-12-02T03:06:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 799931392 bytes 2023-12-02T03:06:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 822301696 bytes 2023-12-02T03:06:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 875173888 bytes 2023-12-02T03:06:49.157 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 895378432 bytes 2023-12-02T03:06:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 952522752 bytes 2023-12-02T03:06:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 979279872 bytes 2023-12-02T03:06:49.496 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr:2023-12-02T03:06:49.493+0000 7f60990ba640 -1 reset not still connected to 0x5611c2db9c70 2023-12-02T03:06:49.496 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr:2023-12-02T03:06:49.493+0000 7f60990ba640 -1 reset not still connected to 0x5611d78a44e0 2023-12-02T03:06:49.562 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1002493952 bytes 2023-12-02T03:06:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1031990272 bytes 2023-12-02T03:06:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1082586112 bytes 2023-12-02T03:06:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1145259008 bytes 2023-12-02T03:06:50.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1210136576 bytes 2023-12-02T03:06:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1268727808 bytes 2023-12-02T03:06:50.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1309519872 bytes 2023-12-02T03:06:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1329680384 bytes 2023-12-02T03:06:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1417989120 bytes 2023-12-02T03:06:50.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1470005248 bytes 2023-12-02T03:06:51.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1509191680 bytes 2023-12-02T03:06:51.170 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1531172864 bytes 2023-12-02T03:06:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1541235712 bytes 2023-12-02T03:06:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1580659712 bytes 2023-12-02T03:06:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1692583936 bytes 2023-12-02T03:06:51.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1732648960 bytes 2023-12-02T03:06:51.858 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1755309056 bytes 2023-12-02T03:06:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1777898496 bytes 2023-12-02T03:06:52.101 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1814392832 bytes 2023-12-02T03:06:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1871601664 bytes 2023-12-02T03:06:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1923162112 bytes 2023-12-02T03:06:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1954027520 bytes 2023-12-02T03:06:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2012971008 bytes 2023-12-02T03:06:52.919 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2091482112 bytes 2023-12-02T03:06:53.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2128512000 bytes 2023-12-02T03:06:53.193 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2173918208 bytes 2023-12-02T03:06:53.329 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2218352640 bytes 2023-12-02T03:06:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2263433216 bytes 2023-12-02T03:06:53.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2308029440 bytes 2023-12-02T03:06:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2308621312 bytes 2023-12-02T03:06:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2341194752 bytes 2023-12-02T03:06:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2360428544 bytes 2023-12-02T03:06:54.096 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2431536128 bytes 2023-12-02T03:06:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2476320768 bytes 2023-12-02T03:06:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2548626432 bytes 2023-12-02T03:06:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2587314176 bytes 2023-12-02T03:06:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2614363136 bytes 2023-12-02T03:06:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2711839744 bytes 2023-12-02T03:06:55.012 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2748284928 bytes 2023-12-02T03:06:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2751311872 bytes 2023-12-02T03:06:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2792450048 bytes 2023-12-02T03:06:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2819783680 bytes 2023-12-02T03:06:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2857238528 bytes 2023-12-02T03:06:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 2888256512 bytes 2023-12-02T03:06:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:06:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:06:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:06:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:06:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:06:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_39 2023-12-02T03:06:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2023-12-02T03:06:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:06:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:06:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_39 2023-12-02T03:06:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:06:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:06:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_39 128M 2023-12-02T03:06:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:06:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:06:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_39 2023-12-02T03:06:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:06:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:06:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:06:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:06:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_39 2023-12-02T03:06:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:06:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:06:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:06:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:06:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2023-12-02T03:06:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:06:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:06:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_39 2023-12-02T03:06:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:06:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2023-12-02T03:06:56.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:06:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2023-12-02T03:06:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_39': 2023-12-02T03:06:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:06:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:06:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:06:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 14389277426b 2023-12-02T03:06:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.14389277426b 2023-12-02T03:06:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:06:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:06:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:06:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:06:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:06:55 2023 2023-12-02T03:06:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:06:55 2023 2023-12-02T03:06:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:06:55 2023 2023-12-02T03:06:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:06:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:06:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: c1ab0b04-68aa-4508-ad3f-25906784bf9e 2023-12-02T03:06:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:06:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:06:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_39 2023-12-02T03:06:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:06:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:06:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_39 2023-12-02T03:06:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:06:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=12 2023-12-02T03:06:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:06:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:06:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 33124352 bytes 2023-12-02T03:06:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 79598592 bytes 2023-12-02T03:06:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 101871616 bytes 2023-12-02T03:06:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 127313920 bytes 2023-12-02T03:06:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 195265536 bytes 2023-12-02T03:06:57.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 268878848 bytes 2023-12-02T03:06:57.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 337031168 bytes 2023-12-02T03:06:58.019 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 359829504 bytes 2023-12-02T03:06:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 368388096 bytes 2023-12-02T03:06:58.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 411167744 bytes 2023-12-02T03:06:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 416513024 bytes 2023-12-02T03:06:58.531 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 491064320 bytes 2023-12-02T03:06:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 523308032 bytes 2023-12-02T03:06:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 578698240 bytes 2023-12-02T03:06:59.000 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 648098816 bytes 2023-12-02T03:06:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 660897792 bytes 2023-12-02T03:06:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 697839616 bytes 2023-12-02T03:06:59.375 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 744781824 bytes 2023-12-02T03:06:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 773891072 bytes 2023-12-02T03:06:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 801045504 bytes 2023-12-02T03:06:59.774 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 875044864 bytes 2023-12-02T03:06:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 899495936 bytes 2023-12-02T03:07:00.069 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 959646720 bytes 2023-12-02T03:07:00.204 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 999916544 bytes 2023-12-02T03:07:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1043120128 bytes 2023-12-02T03:07:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1110384640 bytes 2023-12-02T03:07:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1171536896 bytes 2023-12-02T03:07:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1224921088 bytes 2023-12-02T03:07:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1263054848 bytes 2023-12-02T03:07:01.053 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1265986560 bytes 2023-12-02T03:07:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1346806784 bytes 2023-12-02T03:07:01.320 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1362574336 bytes 2023-12-02T03:07:01.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1382760448 bytes 2023-12-02T03:07:01.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1451033600 bytes 2023-12-02T03:07:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1485385728 bytes 2023-12-02T03:07:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1495824384 bytes 2023-12-02T03:07:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1547145216 bytes 2023-12-02T03:07:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1558641664 bytes 2023-12-02T03:07:02.252 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1618495488 bytes 2023-12-02T03:07:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1665765376 bytes 2023-12-02T03:07:02.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1729675264 bytes 2023-12-02T03:07:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1821443072 bytes 2023-12-02T03:07:02.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1822836736 bytes 2023-12-02T03:07:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1864929280 bytes 2023-12-02T03:07:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1891585024 bytes 2023-12-02T03:07:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1963249664 bytes 2023-12-02T03:07:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 2041817088 bytes 2023-12-02T03:07:03.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2081090560 bytes 2023-12-02T03:07:03.781 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2140071936 bytes 2023-12-02T03:07:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2159627264 bytes 2023-12-02T03:07:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2212507648 bytes 2023-12-02T03:07:04.173 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2256807936 bytes 2023-12-02T03:07:04.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2355018752 bytes 2023-12-02T03:07:04.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2448541696 bytes 2023-12-02T03:07:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2509037568 bytes 2023-12-02T03:07:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2594311168 bytes 2023-12-02T03:07:05.095 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2656133120 bytes 2023-12-02T03:07:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2719899648 bytes 2023-12-02T03:07:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2746217472 bytes 2023-12-02T03:07:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2786830336 bytes 2023-12-02T03:07:05.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2791228416 bytes 2023-12-02T03:07:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2799762432 bytes 2023-12-02T03:07:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2844505088 bytes 2023-12-02T03:07:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2930553856 bytes 2023-12-02T03:07:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2973222912 bytes 2023-12-02T03:07:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 3026588672 bytes 2023-12-02T03:07:06.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 3076083712 bytes 2023-12-02T03:07:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 3113401344 bytes 2023-12-02T03:07:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 3154211840 bytes 2023-12-02T03:07:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 3195637760 bytes 2023-12-02T03:07:07.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 3245645824 bytes 2023-12-02T03:07:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 3307003904 bytes 2023-12-02T03:07:07.346 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 3307624448 bytes 2023-12-02T03:07:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3358898176 bytes 2023-12-02T03:07:07.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3428460544 bytes 2023-12-02T03:07:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3475777536 bytes 2023-12-02T03:07:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3569211392 bytes 2023-12-02T03:07:08.169 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3612516352 bytes 2023-12-02T03:07:08.334 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3679924224 bytes 2023-12-02T03:07:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3699082240 bytes 2023-12-02T03:07:08.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3758020608 bytes 2023-12-02T03:07:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:07:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:07:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:07:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:07:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:07:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_40 2023-12-02T03:07:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2023-12-02T03:07:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:07:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_40 2023-12-02T03:07:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:07:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:07:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_40 128M 2023-12-02T03:07:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:07:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_40 2023-12-02T03:07:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:07:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:07:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:07:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_40 2023-12-02T03:07:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:07:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:07:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:07:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2023-12-02T03:07:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:07:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_40 2023-12-02T03:07:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:07:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2023-12-02T03:07:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:07:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2023-12-02T03:07:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_40': 2023-12-02T03:07:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:07:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:07:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:07:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 1441c6ec7ccb 2023-12-02T03:07:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.1441c6ec7ccb 2023-12-02T03:07:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:07:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:07:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:07:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:07:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:07:08 2023 2023-12-02T03:07:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:07:08 2023 2023-12-02T03:07:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:07:08 2023 2023-12-02T03:07:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:07:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:07:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: f12eeb37-6efd-4241-9808-5371a9611fae 2023-12-02T03:07:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:07:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:07:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_40 2023-12-02T03:07:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:07:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_40 2023-12-02T03:07:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:07:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=12 2023-12-02T03:07:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:07:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:07:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 60697600 bytes 2023-12-02T03:07:10.026 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 113254400 bytes 2023-12-02T03:07:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 135111680 bytes 2023-12-02T03:07:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 186550272 bytes 2023-12-02T03:07:10.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 258702336 bytes 2023-12-02T03:07:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 291021824 bytes 2023-12-02T03:07:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 346902528 bytes 2023-12-02T03:07:10.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 377563136 bytes 2023-12-02T03:07:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 391591936 bytes 2023-12-02T03:07:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 414652416 bytes 2023-12-02T03:07:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 459715584 bytes 2023-12-02T03:07:11.418 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 548304896 bytes 2023-12-02T03:07:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 581259264 bytes 2023-12-02T03:07:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 624594944 bytes 2023-12-02T03:07:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 663720960 bytes 2023-12-02T03:07:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 700597248 bytes 2023-12-02T03:07:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 726061056 bytes 2023-12-02T03:07:12.236 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 794515456 bytes 2023-12-02T03:07:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 849465344 bytes 2023-12-02T03:07:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 862764032 bytes 2023-12-02T03:07:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 928321536 bytes 2023-12-02T03:07:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 1027910656 bytes 2023-12-02T03:07:13.000 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 1054540800 bytes 2023-12-02T03:07:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1129732096 bytes 2023-12-02T03:07:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1201540096 bytes 2023-12-02T03:07:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1208372224 bytes 2023-12-02T03:07:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1232859136 bytes 2023-12-02T03:07:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1267531776 bytes 2023-12-02T03:07:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1341539328 bytes 2023-12-02T03:07:14.033 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8266120312137001) > weight (0.25) 2023-12-02T03:07:14.033 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.768586011592344) > weight (0.25) 2023-12-02T03:07:14.033 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9714096746350961) > weight (0.25) 2023-12-02T03:07:14.033 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-12-02T03:07:14.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-12-02T03:07:14.034 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-02T03:07:14.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi057.stderr:2023-12-02T03:07:14.032+0000 7f9fc727f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 102854) UID: 1000 2023-12-02T03:07:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1390489600 bytes 2023-12-02T03:07:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1403016192 bytes 2023-12-02T03:07:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1431656448 bytes 2023-12-02T03:07:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1490970624 bytes 2023-12-02T03:07:14.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1518592000 bytes 2023-12-02T03:07:14.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1583622144 bytes 2023-12-02T03:07:14.854 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1613250560 bytes 2023-12-02T03:07:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1692298240 bytes 2023-12-02T03:07:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1715497984 bytes 2023-12-02T03:07:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1716110336 bytes 2023-12-02T03:07:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1788604416 bytes 2023-12-02T03:07:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1791233024 bytes 2023-12-02T03:07:15.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1825830912 bytes 2023-12-02T03:07:15.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1860623360 bytes 2023-12-02T03:07:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1885283328 bytes 2023-12-02T03:07:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1976952832 bytes 2023-12-02T03:07:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 2039669760 bytes 2023-12-02T03:07:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 2040290304 bytes 2023-12-02T03:07:16.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2082165760 bytes 2023-12-02T03:07:16.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2160093184 bytes 2023-12-02T03:07:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2232567808 bytes 2023-12-02T03:07:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2268917760 bytes 2023-12-02T03:07:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2334779392 bytes 2023-12-02T03:07:17.411 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2406602752 bytes 2023-12-02T03:07:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2430501888 bytes 2023-12-02T03:07:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2519563264 bytes 2023-12-02T03:07:17.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2538344448 bytes 2023-12-02T03:07:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2582891520 bytes 2023-12-02T03:07:18.165 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2652248064 bytes 2023-12-02T03:07:18.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2681589760 bytes 2023-12-02T03:07:18.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2699310080 bytes 2023-12-02T03:07:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2725835776 bytes 2023-12-02T03:07:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2742475776 bytes 2023-12-02T03:07:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2764827648 bytes 2023-12-02T03:07:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2868384768 bytes 2023-12-02T03:07:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2902544384 bytes 2023-12-02T03:07:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2932613120 bytes 2023-12-02T03:07:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2983496704 bytes 2023-12-02T03:07:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 3029036032 bytes 2023-12-02T03:07:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 3080192000 bytes 2023-12-02T03:07:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 3097950208 bytes 2023-12-02T03:07:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 3111219200 bytes 2023-12-02T03:07:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 3184268288 bytes 2023-12-02T03:07:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 3229981696 bytes 2023-12-02T03:07:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3261672448 bytes 2023-12-02T03:07:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3283970048 bytes 2023-12-02T03:07:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3352844288 bytes 2023-12-02T03:07:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3376953344 bytes 2023-12-02T03:07:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3437203456 bytes 2023-12-02T03:07:21.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3457421312 bytes 2023-12-02T03:07:21.035 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-12-02T03:07:21.035 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T03:07:21.035 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-12-02T03:07:21.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-02T03:07:21.036 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-02T03:07:21.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-02T03:07:21.038 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2023-12-02T03:07:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3462724608 bytes 2023-12-02T03:07:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3488254976 bytes 2023-12-02T03:07:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8200 IOs, 3572762624 bytes 2023-12-02T03:07:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8300 IOs, 3593649152 bytes 2023-12-02T03:07:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:07:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:07:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:07:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:07:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:07:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_41 2023-12-02T03:07:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2023-12-02T03:07:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:07:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_41 2023-12-02T03:07:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:07:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:07:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_41 128M 2023-12-02T03:07:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:07:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_41 2023-12-02T03:07:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:07:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:07:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:07:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_41 2023-12-02T03:07:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:07:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:07:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:07:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2023-12-02T03:07:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:07:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_41 2023-12-02T03:07:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:07:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2023-12-02T03:07:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:07:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2023-12-02T03:07:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_41': 2023-12-02T03:07:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:07:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:07:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:07:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 145074567408 2023-12-02T03:07:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.145074567408 2023-12-02T03:07:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:07:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:07:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:07:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:07:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:07:21 2023 2023-12-02T03:07:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:07:21 2023 2023-12-02T03:07:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:07:21 2023 2023-12-02T03:07:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:07:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:07:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 5db3178e-6670-496c-bd76-373d30e83937 2023-12-02T03:07:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:07:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:07:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_41 2023-12-02T03:07:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:07:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_41 2023-12-02T03:07:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:07:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=6 2023-12-02T03:07:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:07:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:07:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 90176512 bytes 2023-12-02T03:07:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 97793024 bytes 2023-12-02T03:07:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 178219008 bytes 2023-12-02T03:07:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 233669632 bytes 2023-12-02T03:07:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 259574784 bytes 2023-12-02T03:07:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 291540992 bytes 2023-12-02T03:07:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 296509440 bytes 2023-12-02T03:07:23.084 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 362191872 bytes 2023-12-02T03:07:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 380880896 bytes 2023-12-02T03:07:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 449822720 bytes 2023-12-02T03:07:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 488798208 bytes 2023-12-02T03:07:23.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 533487616 bytes 2023-12-02T03:07:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 542595072 bytes 2023-12-02T03:07:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 573281280 bytes 2023-12-02T03:07:23.883 INFO:tasks.ceph.cluster2.osd.2.smithi057.stderr:2023-12-02T03:07:23.880+0000 7f34767b9640 -1 reset not still connected to 0x5622586c3d40 2023-12-02T03:07:23.885 INFO:tasks.ceph.cluster2.osd.2.smithi057.stderr:2023-12-02T03:07:23.882+0000 7f34767b9640 -1 reset not still connected to 0x56225d686f70 2023-12-02T03:07:23.885 INFO:tasks.ceph.cluster2.osd.2.smithi057.stderr:2023-12-02T03:07:23.882+0000 7f34767b9640 -1 reset not still connected to 0x56226172eea0 2023-12-02T03:07:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 584643584 bytes 2023-12-02T03:07:24.102 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 615148544 bytes 2023-12-02T03:07:24.220 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 632355840 bytes 2023-12-02T03:07:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 677385216 bytes 2023-12-02T03:07:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 702127104 bytes 2023-12-02T03:07:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 751903744 bytes 2023-12-02T03:07:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 808394752 bytes 2023-12-02T03:07:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 849876992 bytes 2023-12-02T03:07:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 873047040 bytes 2023-12-02T03:07:25.240 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 966396928 bytes 2023-12-02T03:07:25.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1005033472 bytes 2023-12-02T03:07:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1058694144 bytes 2023-12-02T03:07:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1094578176 bytes 2023-12-02T03:07:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1095187456 bytes 2023-12-02T03:07:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1105143808 bytes 2023-12-02T03:07:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1201249280 bytes 2023-12-02T03:07:26.200 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1238352896 bytes 2023-12-02T03:07:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1296437248 bytes 2023-12-02T03:07:26.523 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1355908096 bytes 2023-12-02T03:07:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1444877312 bytes 2023-12-02T03:07:26.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1552558080 bytes 2023-12-02T03:07:27.058 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1572277248 bytes 2023-12-02T03:07:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1626580992 bytes 2023-12-02T03:07:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1646838784 bytes 2023-12-02T03:07:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1681638400 bytes 2023-12-02T03:07:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1717003264 bytes 2023-12-02T03:07:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1741222912 bytes 2023-12-02T03:07:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1765807104 bytes 2023-12-02T03:07:27.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:07:27.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:07:27.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:07:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:07:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:07:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_42 2023-12-02T03:07:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2023-12-02T03:07:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:07:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_42 2023-12-02T03:07:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:07:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:07:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_42 128M 2023-12-02T03:07:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:07:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_42 2023-12-02T03:07:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:07:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:07:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:07:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_42 2023-12-02T03:07:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:07:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:07:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:07:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2023-12-02T03:07:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:07:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_42 2023-12-02T03:07:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:07:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2023-12-02T03:07:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:07:28.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2023-12-02T03:07:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_42': 2023-12-02T03:07:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:07:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:07:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:07:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 1458a1389144 2023-12-02T03:07:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.1458a1389144 2023-12-02T03:07:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:07:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:07:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:07:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:07:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:07:27 2023 2023-12-02T03:07:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:07:27 2023 2023-12-02T03:07:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:07:27 2023 2023-12-02T03:07:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:07:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:07:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 55cf3916-f72c-47f2-b9d9-b3f8b8496445 2023-12-02T03:07:28.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:07:28.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:07:28.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_42 2023-12-02T03:07:28.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:07:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_42 2023-12-02T03:07:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:07:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=13 2023-12-02T03:07:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:07:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:07:28.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 117474304 bytes 2023-12-02T03:07:29.073 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 155197440 bytes 2023-12-02T03:07:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 189031424 bytes 2023-12-02T03:07:29.374 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 263262208 bytes 2023-12-02T03:07:29.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 269711360 bytes 2023-12-02T03:07:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 303885312 bytes 2023-12-02T03:07:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 327610368 bytes 2023-12-02T03:07:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 410967040 bytes 2023-12-02T03:07:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 466451456 bytes 2023-12-02T03:07:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 521979904 bytes 2023-12-02T03:07:30.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 588703744 bytes 2023-12-02T03:07:30.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 676377600 bytes 2023-12-02T03:07:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 790811648 bytes 2023-12-02T03:07:30.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 848663552 bytes 2023-12-02T03:07:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 853742592 bytes 2023-12-02T03:07:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 870511616 bytes 2023-12-02T03:07:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 886227968 bytes 2023-12-02T03:07:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 913428480 bytes 2023-12-02T03:07:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 936359936 bytes 2023-12-02T03:07:31.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 966243328 bytes 2023-12-02T03:07:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 992467968 bytes 2023-12-02T03:07:31.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 1021897728 bytes 2023-12-02T03:07:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 1022519296 bytes 2023-12-02T03:07:32.056 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1051700224 bytes 2023-12-02T03:07:32.215 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1111430144 bytes 2023-12-02T03:07:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1149881344 bytes 2023-12-02T03:07:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1178031104 bytes 2023-12-02T03:07:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1200937984 bytes 2023-12-02T03:07:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1270142976 bytes 2023-12-02T03:07:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1315534848 bytes 2023-12-02T03:07:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1359828992 bytes 2023-12-02T03:07:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1401112576 bytes 2023-12-02T03:07:33.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1416182784 bytes 2023-12-02T03:07:33.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1435591680 bytes 2023-12-02T03:07:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1473941504 bytes 2023-12-02T03:07:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1552410624 bytes 2023-12-02T03:07:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1569513472 bytes 2023-12-02T03:07:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1623022592 bytes 2023-12-02T03:07:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1623643136 bytes 2023-12-02T03:07:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1633143808 bytes 2023-12-02T03:07:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1650575360 bytes 2023-12-02T03:07:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1719773184 bytes 2023-12-02T03:07:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1780527104 bytes 2023-12-02T03:07:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1816355840 bytes 2023-12-02T03:07:34.962 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1928579072 bytes 2023-12-02T03:07:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1939573760 bytes 2023-12-02T03:07:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 2018668544 bytes 2023-12-02T03:07:35.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2052697088 bytes 2023-12-02T03:07:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2094466048 bytes 2023-12-02T03:07:35.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2120155136 bytes 2023-12-02T03:07:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2188505088 bytes 2023-12-02T03:07:35.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2227775488 bytes 2023-12-02T03:07:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2233762816 bytes 2023-12-02T03:07:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2303752192 bytes 2023-12-02T03:07:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2354330624 bytes 2023-12-02T03:07:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2418899968 bytes 2023-12-02T03:07:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2446810112 bytes 2023-12-02T03:07:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2513060864 bytes 2023-12-02T03:07:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2513664000 bytes 2023-12-02T03:07:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2599998464 bytes 2023-12-02T03:07:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2600602624 bytes 2023-12-02T03:07:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2627555328 bytes 2023-12-02T03:07:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2709851136 bytes 2023-12-02T03:07:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2738589696 bytes 2023-12-02T03:07:37.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2779640832 bytes 2023-12-02T03:07:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2890254336 bytes 2023-12-02T03:07:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2924490752 bytes 2023-12-02T03:07:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2969234432 bytes 2023-12-02T03:07:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 3042343936 bytes 2023-12-02T03:07:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 3081789440 bytes 2023-12-02T03:07:38.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 3148929024 bytes 2023-12-02T03:07:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 3194143744 bytes 2023-12-02T03:07:39.022 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 3249568768 bytes 2023-12-02T03:07:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3338443776 bytes 2023-12-02T03:07:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3417838592 bytes 2023-12-02T03:07:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3464725504 bytes 2023-12-02T03:07:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3515086848 bytes 2023-12-02T03:07:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3606703104 bytes 2023-12-02T03:07:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3650822144 bytes 2023-12-02T03:07:40.244 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3722733568 bytes 2023-12-02T03:07:40.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3790763008 bytes 2023-12-02T03:07:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8200 IOs, 3807449088 bytes 2023-12-02T03:07:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8300 IOs, 3853924352 bytes 2023-12-02T03:07:40.778 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8400 IOs, 3874000896 bytes 2023-12-02T03:07:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8500 IOs, 3876746240 bytes 2023-12-02T03:07:41.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8600 IOs, 3926740992 bytes 2023-12-02T03:07:41.097 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8700 IOs, 3927382016 bytes 2023-12-02T03:07:41.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8800 IOs, 4024723456 bytes 2023-12-02T03:07:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8900 IOs, 4057124864 bytes 2023-12-02T03:07:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9000 IOs, 4097776640 bytes 2023-12-02T03:07:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:07:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:07:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:07:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:07:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:07:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_43 2023-12-02T03:07:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2023-12-02T03:07:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:07:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_43 2023-12-02T03:07:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:07:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:07:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_43 128M 2023-12-02T03:07:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:07:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_43 2023-12-02T03:07:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:07:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:07:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:07:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_43 2023-12-02T03:07:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:07:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:07:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:07:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2023-12-02T03:07:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:07:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_43 2023-12-02T03:07:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:07:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2023-12-02T03:07:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:07:41.856 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2023-12-02T03:07:41.892 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_43': 2023-12-02T03:07:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:07:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:07:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:07:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 1463c67a3e28 2023-12-02T03:07:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.1463c67a3e28 2023-12-02T03:07:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:07:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:07:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:07:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:07:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:07:41 2023 2023-12-02T03:07:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:07:41 2023 2023-12-02T03:07:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:07:41 2023 2023-12-02T03:07:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:07:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:07:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 0c9ffe84-d24c-4f74-b087-b033f2cc4422 2023-12-02T03:07:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:07:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:07:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_43 2023-12-02T03:07:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:07:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_43 2023-12-02T03:07:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:07:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=10 2023-12-02T03:07:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:07:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:07:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 27817984 bytes 2023-12-02T03:07:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 35606528 bytes 2023-12-02T03:07:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 46380032 bytes 2023-12-02T03:07:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 92858368 bytes 2023-12-02T03:07:42.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 127754240 bytes 2023-12-02T03:07:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 160840704 bytes 2023-12-02T03:07:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 239079424 bytes 2023-12-02T03:07:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 268295168 bytes 2023-12-02T03:07:43.057 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 290993152 bytes 2023-12-02T03:07:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 361182208 bytes 2023-12-02T03:07:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 413336576 bytes 2023-12-02T03:07:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 435508224 bytes 2023-12-02T03:07:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 456985600 bytes 2023-12-02T03:07:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 486900736 bytes 2023-12-02T03:07:43.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 532795392 bytes 2023-12-02T03:07:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 565684224 bytes 2023-12-02T03:07:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 615076864 bytes 2023-12-02T03:07:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 632943616 bytes 2023-12-02T03:07:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 654467072 bytes 2023-12-02T03:07:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 748931072 bytes 2023-12-02T03:07:44.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 785810432 bytes 2023-12-02T03:07:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 856578048 bytes 2023-12-02T03:07:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 916339712 bytes 2023-12-02T03:07:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 996985856 bytes 2023-12-02T03:07:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1037812736 bytes 2023-12-02T03:07:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1100739584 bytes 2023-12-02T03:07:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1125347328 bytes 2023-12-02T03:07:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1176907776 bytes 2023-12-02T03:07:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1195944960 bytes 2023-12-02T03:07:46.102 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1207176192 bytes 2023-12-02T03:07:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1267566592 bytes 2023-12-02T03:07:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1297985536 bytes 2023-12-02T03:07:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1368898560 bytes 2023-12-02T03:07:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1383083008 bytes 2023-12-02T03:07:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1458322432 bytes 2023-12-02T03:07:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1538694144 bytes 2023-12-02T03:07:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1665225728 bytes 2023-12-02T03:07:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1725962240 bytes 2023-12-02T03:07:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1737539584 bytes 2023-12-02T03:07:47.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1738150912 bytes 2023-12-02T03:07:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1791138816 bytes 2023-12-02T03:07:47.962 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1823753216 bytes 2023-12-02T03:07:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1850762240 bytes 2023-12-02T03:07:48.240 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1895924736 bytes 2023-12-02T03:07:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1923991552 bytes 2023-12-02T03:07:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1991770112 bytes 2023-12-02T03:07:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 2014446592 bytes 2023-12-02T03:07:48.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2080351232 bytes 2023-12-02T03:07:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2152464384 bytes 2023-12-02T03:07:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2153084928 bytes 2023-12-02T03:07:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2188382208 bytes 2023-12-02T03:07:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2252040192 bytes 2023-12-02T03:07:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2316308480 bytes 2023-12-02T03:07:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2365411328 bytes 2023-12-02T03:07:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2373518336 bytes 2023-12-02T03:07:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2419178496 bytes 2023-12-02T03:07:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2447582208 bytes 2023-12-02T03:07:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2513500160 bytes 2023-12-02T03:07:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2565963776 bytes 2023-12-02T03:07:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2614645760 bytes 2023-12-02T03:07:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2644145152 bytes 2023-12-02T03:07:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2652071936 bytes 2023-12-02T03:07:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2685035520 bytes 2023-12-02T03:07:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2732524544 bytes 2023-12-02T03:07:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2769816576 bytes 2023-12-02T03:07:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2840195072 bytes 2023-12-02T03:07:51.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2856274944 bytes 2023-12-02T03:07:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2903502848 bytes 2023-12-02T03:07:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 2921628672 bytes 2023-12-02T03:07:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 2948358144 bytes 2023-12-02T03:07:51.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:07:51.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:07:51.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:07:51.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:07:51.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:07:51.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_44 2023-12-02T03:07:51.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2023-12-02T03:07:51.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:07:51.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:51.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:51.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:51.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_44 2023-12-02T03:07:51.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:51.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:07:51.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:07:51.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:51.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_44 128M 2023-12-02T03:07:51.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:07:51.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:51.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:51.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:51.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_44 2023-12-02T03:07:51.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:07:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:07:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:07:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_44 2023-12-02T03:07:51.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:07:51.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:51.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:07:51.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:07:51.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2023-12-02T03:07:51.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:07:51.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_44 2023-12-02T03:07:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:07:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2023-12-02T03:07:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:07:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2023-12-02T03:07:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_44': 2023-12-02T03:07:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:07:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:07:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:07:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 146ca24dff32 2023-12-02T03:07:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.146ca24dff32 2023-12-02T03:07:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:07:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:07:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:07:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:07:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:07:51 2023 2023-12-02T03:07:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:07:51 2023 2023-12-02T03:07:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:07:51 2023 2023-12-02T03:07:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:07:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:07:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 2e4177a2-c078-4cd9-b5d2-ee2477a02290 2023-12-02T03:07:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:07:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:07:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_44 2023-12-02T03:07:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:07:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_44 2023-12-02T03:07:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:07:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=6 2023-12-02T03:07:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:07:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:07:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 111923200 bytes 2023-12-02T03:07:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 164590592 bytes 2023-12-02T03:07:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 203166720 bytes 2023-12-02T03:07:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 235710464 bytes 2023-12-02T03:07:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 282012672 bytes 2023-12-02T03:07:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 325532672 bytes 2023-12-02T03:07:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 389095424 bytes 2023-12-02T03:07:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 472867840 bytes 2023-12-02T03:07:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 582770688 bytes 2023-12-02T03:07:53.814 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 694051840 bytes 2023-12-02T03:07:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 732549120 bytes 2023-12-02T03:07:54.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 765107200 bytes 2023-12-02T03:07:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 821920768 bytes 2023-12-02T03:07:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 855055360 bytes 2023-12-02T03:07:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 911945728 bytes 2023-12-02T03:07:54.692 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 958692352 bytes 2023-12-02T03:07:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 971691008 bytes 2023-12-02T03:07:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 1067628544 bytes 2023-12-02T03:07:55.117 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 1090846720 bytes 2023-12-02T03:07:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 1123286016 bytes 2023-12-02T03:07:55.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 1153844224 bytes 2023-12-02T03:07:55.548 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 1226622976 bytes 2023-12-02T03:07:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 1264273408 bytes 2023-12-02T03:07:55.809 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1305089024 bytes 2023-12-02T03:07:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1364737024 bytes 2023-12-02T03:07:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1434951680 bytes 2023-12-02T03:07:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1486065664 bytes 2023-12-02T03:07:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1535566848 bytes 2023-12-02T03:07:56.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1564779520 bytes 2023-12-02T03:07:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1607699456 bytes 2023-12-02T03:07:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1710717952 bytes 2023-12-02T03:07:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1737751552 bytes 2023-12-02T03:07:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1793166336 bytes 2023-12-02T03:07:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1828651008 bytes 2023-12-02T03:07:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1864196096 bytes 2023-12-02T03:07:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1917251584 bytes 2023-12-02T03:07:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1958864896 bytes 2023-12-02T03:07:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1962374144 bytes 2023-12-02T03:07:57.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1995728896 bytes 2023-12-02T03:07:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:07:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:07:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:07:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:07:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:07:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_45 2023-12-02T03:07:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2023-12-02T03:07:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:07:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_45 2023-12-02T03:07:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:07:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:07:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_45 128M 2023-12-02T03:07:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:07:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_45 2023-12-02T03:07:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:07:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:07:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:07:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:07:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_45 2023-12-02T03:07:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:07:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:07:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:07:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2023-12-02T03:07:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:07:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_45 2023-12-02T03:07:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:07:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2023-12-02T03:07:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:07:59.225 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2023-12-02T03:07:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_45': 2023-12-02T03:07:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:07:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:07:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:07:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 14761488423e 2023-12-02T03:07:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.14761488423e 2023-12-02T03:07:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:07:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:07:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:07:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:07:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:07:58 2023 2023-12-02T03:07:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:07:58 2023 2023-12-02T03:07:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:07:58 2023 2023-12-02T03:07:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:07:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:07:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: c46da9c3-6d9c-4389-a394-a73d56fb3cdf 2023-12-02T03:07:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:07:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:07:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_45 2023-12-02T03:07:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:07:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:07:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_45 2023-12-02T03:07:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:07:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=11 2023-12-02T03:07:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:07:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:07:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 66452480 bytes 2023-12-02T03:07:59.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 71217152 bytes 2023-12-02T03:07:59.779 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 186507264 bytes 2023-12-02T03:07:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 229328896 bytes 2023-12-02T03:08:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 265815040 bytes 2023-12-02T03:08:00.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 282258432 bytes 2023-12-02T03:08:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 286654464 bytes 2023-12-02T03:08:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 315244544 bytes 2023-12-02T03:08:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 324990976 bytes 2023-12-02T03:08:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 356747264 bytes 2023-12-02T03:08:00.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 400818176 bytes 2023-12-02T03:08:00.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 505564160 bytes 2023-12-02T03:08:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 564984832 bytes 2023-12-02T03:08:01.296 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 627086336 bytes 2023-12-02T03:08:01.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 675599360 bytes 2023-12-02T03:08:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 692605952 bytes 2023-12-02T03:08:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 712654848 bytes 2023-12-02T03:08:01.784 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 747810816 bytes 2023-12-02T03:08:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 775221248 bytes 2023-12-02T03:08:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 789347328 bytes 2023-12-02T03:08:02.172 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 822491136 bytes 2023-12-02T03:08:02.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 852572160 bytes 2023-12-02T03:08:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 887976960 bytes 2023-12-02T03:08:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 923991040 bytes 2023-12-02T03:08:02.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 965820416 bytes 2023-12-02T03:08:02.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 984675328 bytes 2023-12-02T03:08:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1017369600 bytes 2023-12-02T03:08:02.959 INFO:tasks.ceph.cluster2.osd.2.smithi057.stderr:2023-12-02T03:08:02.956+0000 7f34767b9640 -1 reset not still connected to 0x5622716bfe10 2023-12-02T03:08:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1060619264 bytes 2023-12-02T03:08:03.242 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1114678272 bytes 2023-12-02T03:08:03.331 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1117092864 bytes 2023-12-02T03:08:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1147052032 bytes 2023-12-02T03:08:03.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1151921152 bytes 2023-12-02T03:08:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1153062912 bytes 2023-12-02T03:08:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1218422784 bytes 2023-12-02T03:08:03.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1277511680 bytes 2023-12-02T03:08:04.120 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1322842112 bytes 2023-12-02T03:08:04.232 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1341324288 bytes 2023-12-02T03:08:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1398546432 bytes 2023-12-02T03:08:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1462815744 bytes 2023-12-02T03:08:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1485165568 bytes 2023-12-02T03:08:04.798 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1528586240 bytes 2023-12-02T03:08:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1548829696 bytes 2023-12-02T03:08:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1573518336 bytes 2023-12-02T03:08:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1642967040 bytes 2023-12-02T03:08:05.319 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1655507968 bytes 2023-12-02T03:08:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1697628160 bytes 2023-12-02T03:08:05.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1738155008 bytes 2023-12-02T03:08:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 1769752576 bytes 2023-12-02T03:08:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 1838128128 bytes 2023-12-02T03:08:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 1893439488 bytes 2023-12-02T03:08:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 1905925120 bytes 2023-12-02T03:08:06.263 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr:2023-12-02T03:08:06.260+0000 7f609cc9b640 -1 bdev(0x5611bf78ae00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-02T03:08:06.265 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7631-gbed76814/rpm/el9/BUILD/ceph-18.0.0-7631-gbed76814/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f609cc9b640 time 2023-12-02T03:08:06.262126+0000 2023-12-02T03:08:06.265 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7631-gbed76814/rpm/el9/BUILD/ceph-18.0.0-7631-gbed76814/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-02T03:08:06.265 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev) 2023-12-02T03:08:06.265 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x5611bd197e7b] 2023-12-02T03:08:06.265 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 2: (KernelDevice::_aio_thread()+0x10f6) [0x5611bdae6d06] 2023-12-02T03:08:06.265 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 3: ceph-osd(+0xd9fef1) [0x5611bdac7ef1] 2023-12-02T03:08:06.265 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7f60a9a9f802] 2023-12-02T03:08:06.265 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7f60a9a3f450] 2023-12-02T03:08:06.265 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr:*** Caught signal (Aborted) ** 2023-12-02T03:08:06.266 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: in thread 7f609cc9b640 thread_name:bstore_aio 2023-12-02T03:08:06.266 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr:2023-12-02T03:08:06.262+0000 7f609cc9b640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7631-gbed76814/rpm/el9/BUILD/ceph-18.0.0-7631-gbed76814/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f609cc9b640 time 2023-12-02T03:08:06.262126+0000 2023-12-02T03:08:06.266 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7631-gbed76814/rpm/el9/BUILD/ceph-18.0.0-7631-gbed76814/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-02T03:08:06.266 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 2023-12-02T03:08:06.266 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev) 2023-12-02T03:08:06.266 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x5611bd197e7b] 2023-12-02T03:08:06.266 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 2: (KernelDevice::_aio_thread()+0x10f6) [0x5611bdae6d06] 2023-12-02T03:08:06.266 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 3: ceph-osd(+0xd9fef1) [0x5611bdac7ef1] 2023-12-02T03:08:06.266 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7f60a9a9f802] 2023-12-02T03:08:06.266 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7f60a9a3f450] 2023-12-02T03:08:06.266 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 2023-12-02T03:08:06.267 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev) 2023-12-02T03:08:06.267 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f60a9a54db0] 2023-12-02T03:08:06.267 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f60a9aa154c] 2023-12-02T03:08:06.267 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 3: raise() 2023-12-02T03:08:06.267 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 4: abort() 2023-12-02T03:08:06.267 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x5611bd197f3c] 2023-12-02T03:08:06.267 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 6: (KernelDevice::_aio_thread()+0x10f6) [0x5611bdae6d06] 2023-12-02T03:08:06.267 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 7: ceph-osd(+0xd9fef1) [0x5611bdac7ef1] 2023-12-02T03:08:06.267 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f60a9a9f802] 2023-12-02T03:08:06.267 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f60a9a3f450] 2023-12-02T03:08:06.267 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr:2023-12-02T03:08:06.264+0000 7f609cc9b640 -1 *** Caught signal (Aborted) ** 2023-12-02T03:08:06.268 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: in thread 7f609cc9b640 thread_name:bstore_aio 2023-12-02T03:08:06.268 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 2023-12-02T03:08:06.268 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev) 2023-12-02T03:08:06.268 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f60a9a54db0] 2023-12-02T03:08:06.268 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f60a9aa154c] 2023-12-02T03:08:06.268 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 3: raise() 2023-12-02T03:08:06.268 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 4: abort() 2023-12-02T03:08:06.268 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x5611bd197f3c] 2023-12-02T03:08:06.268 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 6: (KernelDevice::_aio_thread()+0x10f6) [0x5611bdae6d06] 2023-12-02T03:08:06.268 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 7: ceph-osd(+0xd9fef1) [0x5611bdac7ef1] 2023-12-02T03:08:06.268 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f60a9a9f802] 2023-12-02T03:08:06.268 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f60a9a3f450] 2023-12-02T03:08:06.268 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-02T03:08:06.269 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 2023-12-02T03:08:06.282 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: -1236> 2023-12-02T03:08:06.260+0000 7f609cc9b640 -1 bdev(0x5611bf78ae00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-02T03:08:06.282 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: -692> 2023-12-02T03:08:06.262+0000 7f609cc9b640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7631-gbed76814/rpm/el9/BUILD/ceph-18.0.0-7631-gbed76814/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f609cc9b640 time 2023-12-02T03:08:06.262126+0000 2023-12-02T03:08:06.282 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7631-gbed76814/rpm/el9/BUILD/ceph-18.0.0-7631-gbed76814/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-02T03:08:06.282 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 2023-12-02T03:08:06.282 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev) 2023-12-02T03:08:06.283 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x5611bd197e7b] 2023-12-02T03:08:06.283 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 2: (KernelDevice::_aio_thread()+0x10f6) [0x5611bdae6d06] 2023-12-02T03:08:06.283 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 3: ceph-osd(+0xd9fef1) [0x5611bdac7ef1] 2023-12-02T03:08:06.283 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7f60a9a9f802] 2023-12-02T03:08:06.283 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7f60a9a3f450] 2023-12-02T03:08:06.283 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 2023-12-02T03:08:06.283 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: -56> 2023-12-02T03:08:06.264+0000 7f609cc9b640 -1 *** Caught signal (Aborted) ** 2023-12-02T03:08:06.283 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: in thread 7f609cc9b640 thread_name:bstore_aio 2023-12-02T03:08:06.283 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 2023-12-02T03:08:06.284 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev) 2023-12-02T03:08:06.284 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f60a9a54db0] 2023-12-02T03:08:06.284 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f60a9aa154c] 2023-12-02T03:08:06.284 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 3: raise() 2023-12-02T03:08:06.284 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 4: abort() 2023-12-02T03:08:06.284 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x5611bd197f3c] 2023-12-02T03:08:06.284 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 6: (KernelDevice::_aio_thread()+0x10f6) [0x5611bdae6d06] 2023-12-02T03:08:06.284 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 7: ceph-osd(+0xd9fef1) [0x5611bdac7ef1] 2023-12-02T03:08:06.284 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f60a9a9f802] 2023-12-02T03:08:06.284 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f60a9a3f450] 2023-12-02T03:08:06.284 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-02T03:08:06.284 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 2023-12-02T03:08:06.291 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: -1655> 2023-12-02T03:08:06.260+0000 7f609cc9b640 -1 bdev(0x5611bf78ae00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-02T03:08:06.291 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: -1111> 2023-12-02T03:08:06.262+0000 7f609cc9b640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7631-gbed76814/rpm/el9/BUILD/ceph-18.0.0-7631-gbed76814/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f609cc9b640 time 2023-12-02T03:08:06.262126+0000 2023-12-02T03:08:06.291 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7631-gbed76814/rpm/el9/BUILD/ceph-18.0.0-7631-gbed76814/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-02T03:08:06.291 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 2023-12-02T03:08:06.291 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev) 2023-12-02T03:08:06.291 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x5611bd197e7b] 2023-12-02T03:08:06.291 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 2: (KernelDevice::_aio_thread()+0x10f6) [0x5611bdae6d06] 2023-12-02T03:08:06.291 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 3: ceph-osd(+0xd9fef1) [0x5611bdac7ef1] 2023-12-02T03:08:06.291 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7f60a9a9f802] 2023-12-02T03:08:06.292 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7f60a9a3f450] 2023-12-02T03:08:06.292 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 2023-12-02T03:08:06.292 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: -475> 2023-12-02T03:08:06.264+0000 7f609cc9b640 -1 *** Caught signal (Aborted) ** 2023-12-02T03:08:06.292 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: in thread 7f609cc9b640 thread_name:bstore_aio 2023-12-02T03:08:06.293 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 2023-12-02T03:08:06.293 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev) 2023-12-02T03:08:06.293 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f60a9a54db0] 2023-12-02T03:08:06.293 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f60a9aa154c] 2023-12-02T03:08:06.293 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 3: raise() 2023-12-02T03:08:06.293 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 4: abort() 2023-12-02T03:08:06.293 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x5611bd197f3c] 2023-12-02T03:08:06.293 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 6: (KernelDevice::_aio_thread()+0x10f6) [0x5611bdae6d06] 2023-12-02T03:08:06.293 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 7: ceph-osd(+0xd9fef1) [0x5611bdac7ef1] 2023-12-02T03:08:06.293 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f60a9a9f802] 2023-12-02T03:08:06.294 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f60a9a3f450] 2023-12-02T03:08:06.294 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-02T03:08:06.294 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr: 2023-12-02T03:08:06.479 INFO:tasks.ceph.cluster2.osd.0.smithi057.stderr:daemon-helper: command failed with exit status 5 2023-12-02T03:08:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 1985925120 bytes 2023-12-02T03:08:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2027004928 bytes 2023-12-02T03:08:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2110833664 bytes 2023-12-02T03:08:08.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2134539264 bytes 2023-12-02T03:08:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2182933504 bytes 2023-12-02T03:08:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2228319232 bytes 2023-12-02T03:08:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2305513472 bytes 2023-12-02T03:08:09.061 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2339059712 bytes 2023-12-02T03:08:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2381282304 bytes 2023-12-02T03:08:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2490639360 bytes 2023-12-02T03:08:09.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2536161280 bytes 2023-12-02T03:08:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2564514816 bytes 2023-12-02T03:08:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2613728256 bytes 2023-12-02T03:08:09.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2700021760 bytes 2023-12-02T03:08:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2777876480 bytes 2023-12-02T03:08:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2819265536 bytes 2023-12-02T03:08:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2828340224 bytes 2023-12-02T03:08:10.046 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 2846635008 bytes 2023-12-02T03:08:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 2865955840 bytes 2023-12-02T03:08:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 2909134848 bytes 2023-12-02T03:08:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:08:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:08:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:08:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:08:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:08:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_46 2023-12-02T03:08:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2023-12-02T03:08:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:08:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:08:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_46 2023-12-02T03:08:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:08:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:08:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_46 128M 2023-12-02T03:08:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:08:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:08:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_46 2023-12-02T03:08:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:08:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:08:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:08:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_46 2023-12-02T03:08:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:08:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:08:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:08:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:08:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2023-12-02T03:08:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:08:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:08:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_46 2023-12-02T03:08:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:08:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2023-12-02T03:08:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:08:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2023-12-02T03:08:10.674 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2023-12-02T03:08:10.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_46': 2023-12-02T03:08:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:08:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:08:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:08:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 147fe3161cd6 2023-12-02T03:08:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.147fe3161cd6 2023-12-02T03:08:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:08:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:08:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:08:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:08:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:08:10 2023 2023-12-02T03:08:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:08:10 2023 2023-12-02T03:08:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:08:10 2023 2023-12-02T03:08:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:08:10.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:08:10.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 0a3b7484-3d9a-411e-9ed7-b9ca249bd93a 2023-12-02T03:08:10.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:08:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:08:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_46 2023-12-02T03:08:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:08:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:08:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_46 2023-12-02T03:08:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:08:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=13 2023-12-02T03:08:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:08:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:08:10.811 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 6429696 bytes 2023-12-02T03:08:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 14407680 bytes 2023-12-02T03:08:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 34501632 bytes 2023-12-02T03:08:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 48931840 bytes 2023-12-02T03:08:11.165 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 148085760 bytes 2023-12-02T03:08:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 206317568 bytes 2023-12-02T03:08:11.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 245720064 bytes 2023-12-02T03:08:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 306510848 bytes 2023-12-02T03:08:11.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 346269696 bytes 2023-12-02T03:08:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 374901760 bytes 2023-12-02T03:08:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 460999680 bytes 2023-12-02T03:08:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 491762688 bytes 2023-12-02T03:08:11.983 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 534299648 bytes 2023-12-02T03:08:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 545302528 bytes 2023-12-02T03:08:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 576324608 bytes 2023-12-02T03:08:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 579735552 bytes 2023-12-02T03:08:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 619235328 bytes 2023-12-02T03:08:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 659599360 bytes 2023-12-02T03:08:12.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 741023744 bytes 2023-12-02T03:08:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 800507904 bytes 2023-12-02T03:08:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 812348416 bytes 2023-12-02T03:08:12.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 838009856 bytes 2023-12-02T03:08:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 888047616 bytes 2023-12-02T03:08:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 961559552 bytes 2023-12-02T03:08:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1026624512 bytes 2023-12-02T03:08:13.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1071971328 bytes 2023-12-02T03:08:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1094217728 bytes 2023-12-02T03:08:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1163682816 bytes 2023-12-02T03:08:13.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1179417600 bytes 2023-12-02T03:08:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1220956160 bytes 2023-12-02T03:08:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1252941824 bytes 2023-12-02T03:08:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1299606528 bytes 2023-12-02T03:08:13.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1367228416 bytes 2023-12-02T03:08:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1404490752 bytes 2023-12-02T03:08:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1443631104 bytes 2023-12-02T03:08:14.262 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1463110656 bytes 2023-12-02T03:08:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1492803584 bytes 2023-12-02T03:08:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1514132480 bytes 2023-12-02T03:08:14.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1589074944 bytes 2023-12-02T03:08:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1635724288 bytes 2023-12-02T03:08:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1697490944 bytes 2023-12-02T03:08:14.872 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1750038528 bytes 2023-12-02T03:08:14.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1768213504 bytes 2023-12-02T03:08:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1816103936 bytes 2023-12-02T03:08:15.167 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1859538944 bytes 2023-12-02T03:08:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1909563392 bytes 2023-12-02T03:08:15.359 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1947531264 bytes 2023-12-02T03:08:15.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 1966108672 bytes 2023-12-02T03:08:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2001905664 bytes 2023-12-02T03:08:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2040398848 bytes 2023-12-02T03:08:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2091112448 bytes 2023-12-02T03:08:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2133575680 bytes 2023-12-02T03:08:15.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2162507776 bytes 2023-12-02T03:08:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2181334016 bytes 2023-12-02T03:08:16.076 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~5s 2023-12-02T03:08:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2273849344 bytes 2023-12-02T03:08:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2296660992 bytes 2023-12-02T03:08:16.370 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2408485888 bytes 2023-12-02T03:08:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2473947136 bytes 2023-12-02T03:08:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2474573824 bytes 2023-12-02T03:08:16.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2550635520 bytes 2023-12-02T03:08:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2609515520 bytes 2023-12-02T03:08:16.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2674698240 bytes 2023-12-02T03:08:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2675305472 bytes 2023-12-02T03:08:17.086 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2716193792 bytes 2023-12-02T03:08:17.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2751022080 bytes 2023-12-02T03:08:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2818149376 bytes 2023-12-02T03:08:17.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2861873152 bytes 2023-12-02T03:08:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2930962432 bytes 2023-12-02T03:08:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 3040291840 bytes 2023-12-02T03:08:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 3040903168 bytes 2023-12-02T03:08:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 3127922688 bytes 2023-12-02T03:08:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 3155594240 bytes 2023-12-02T03:08:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 3211409408 bytes 2023-12-02T03:08:18.173 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3234696192 bytes 2023-12-02T03:08:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3286396928 bytes 2023-12-02T03:08:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3287009280 bytes 2023-12-02T03:08:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3329847296 bytes 2023-12-02T03:08:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3398380544 bytes 2023-12-02T03:08:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3494770688 bytes 2023-12-02T03:08:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3506483200 bytes 2023-12-02T03:08:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3561225216 bytes 2023-12-02T03:08:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8200 IOs, 3603235840 bytes 2023-12-02T03:08:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8300 IOs, 3621387264 bytes 2023-12-02T03:08:19.162 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8400 IOs, 3622006784 bytes 2023-12-02T03:08:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8500 IOs, 3635414016 bytes 2023-12-02T03:08:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8600 IOs, 3644472320 bytes 2023-12-02T03:08:19.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8700 IOs, 3699609600 bytes 2023-12-02T03:08:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8800 IOs, 3713735680 bytes 2023-12-02T03:08:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8900 IOs, 3742366720 bytes 2023-12-02T03:08:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9000 IOs, 3801856000 bytes 2023-12-02T03:08:19.823 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9100 IOs, 3831273472 bytes 2023-12-02T03:08:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9200 IOs, 3888408576 bytes 2023-12-02T03:08:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9300 IOs, 3953397760 bytes 2023-12-02T03:08:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9400 IOs, 4009857024 bytes 2023-12-02T03:08:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9500 IOs, 4024803328 bytes 2023-12-02T03:08:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9600 IOs, 4108978176 bytes 2023-12-02T03:08:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9700 IOs, 4127253504 bytes 2023-12-02T03:08:20.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9800 IOs, 4162852864 bytes 2023-12-02T03:08:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9900 IOs, 4206425088 bytes 2023-12-02T03:08:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10000 IOs, 4288777216 bytes 2023-12-02T03:08:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10100 IOs, 4295430144 bytes 2023-12-02T03:08:20.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10200 IOs, 4332789760 bytes 2023-12-02T03:08:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10300 IOs, 4355957760 bytes 2023-12-02T03:08:21.104 INFO:tasks.ceph.cluster2.osd.2.smithi057.stderr:2023-12-02T03:08:21.101+0000 7f34767b9640 -1 reset not still connected to 0x56225c500680 2023-12-02T03:08:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10400 IOs, 4394248192 bytes 2023-12-02T03:08:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10500 IOs, 4410743808 bytes 2023-12-02T03:08:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10600 IOs, 4447665152 bytes 2023-12-02T03:08:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10700 IOs, 4495596544 bytes 2023-12-02T03:08:21.481 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~11s 2023-12-02T03:08:21.492 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10800 IOs, 4504853504 bytes 2023-12-02T03:08:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10900 IOs, 4528361472 bytes 2023-12-02T03:08:21.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11000 IOs, 4593726464 bytes 2023-12-02T03:08:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11100 IOs, 4610166784 bytes 2023-12-02T03:08:21.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11200 IOs, 4630700032 bytes 2023-12-02T03:08:21.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11300 IOs, 4676529152 bytes 2023-12-02T03:08:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11400 IOs, 4677394432 bytes 2023-12-02T03:08:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11500 IOs, 4708416512 bytes 2023-12-02T03:08:22.203 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11600 IOs, 4773095424 bytes 2023-12-02T03:08:22.334 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11700 IOs, 4831043584 bytes 2023-12-02T03:08:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11800 IOs, 4855510016 bytes 2023-12-02T03:08:22.559 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11900 IOs, 4974050304 bytes 2023-12-02T03:08:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:12000 IOs, 5033854976 bytes 2023-12-02T03:08:22.796 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:12100 IOs, 5099131904 bytes 2023-12-02T03:08:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:12200 IOs, 5121281024 bytes 2023-12-02T03:08:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:12300 IOs, 5218371584 bytes 2023-12-02T03:08:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:12400 IOs, 5258897408 bytes 2023-12-02T03:08:23.224 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:12500 IOs, 5314158592 bytes 2023-12-02T03:08:23.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:12600 IOs, 5394410496 bytes 2023-12-02T03:08:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:12700 IOs, 5419219968 bytes 2023-12-02T03:08:23.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:12800 IOs, 5512524800 bytes 2023-12-02T03:08:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:08:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:08:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:08:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:08:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:08:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_47 2023-12-02T03:08:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2023-12-02T03:08:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:08:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:08:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_47 2023-12-02T03:08:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:08:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:08:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_47 128M 2023-12-02T03:08:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:08:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:08:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_47 2023-12-02T03:08:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:08:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:08:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:08:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_47 2023-12-02T03:08:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:08:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:08:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:08:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:08:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2023-12-02T03:08:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:08:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:08:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_47 2023-12-02T03:08:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:08:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2023-12-02T03:08:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:08:24.218 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2023-12-02T03:08:24.258 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_47': 2023-12-02T03:08:24.258 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:08:24.258 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:08:24.258 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:08:24.258 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 1488d813f206 2023-12-02T03:08:24.258 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.1488d813f206 2023-12-02T03:08:24.258 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:08:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:08:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:08:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:08:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:08:23 2023 2023-12-02T03:08:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:08:23 2023 2023-12-02T03:08:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:08:23 2023 2023-12-02T03:08:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:08:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:08:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: be8f73e3-45b7-4b7e-8479-7fd35126d17d 2023-12-02T03:08:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:08:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:08:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_47 2023-12-02T03:08:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:08:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:08:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_47 2023-12-02T03:08:24.262 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:08:24.263 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=8 2023-12-02T03:08:24.263 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:08:24.263 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:08:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 5908480 bytes 2023-12-02T03:08:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 84561920 bytes 2023-12-02T03:08:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 163701760 bytes 2023-12-02T03:08:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 221030400 bytes 2023-12-02T03:08:24.784 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 245938176 bytes 2023-12-02T03:08:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 254367744 bytes 2023-12-02T03:08:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 304708608 bytes 2023-12-02T03:08:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 317034496 bytes 2023-12-02T03:08:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 339286016 bytes 2023-12-02T03:08:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 383182848 bytes 2023-12-02T03:08:25.275 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 406459392 bytes 2023-12-02T03:08:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 452076544 bytes 2023-12-02T03:08:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 481462272 bytes 2023-12-02T03:08:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 568075264 bytes 2023-12-02T03:08:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 630150144 bytes 2023-12-02T03:08:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 659867648 bytes 2023-12-02T03:08:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 693741568 bytes 2023-12-02T03:08:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 732919808 bytes 2023-12-02T03:08:26.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 766073856 bytes 2023-12-02T03:08:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 865053696 bytes 2023-12-02T03:08:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 898241536 bytes 2023-12-02T03:08:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 915719168 bytes 2023-12-02T03:08:26.461 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 954024960 bytes 2023-12-02T03:08:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 985507840 bytes 2023-12-02T03:08:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 998872064 bytes 2023-12-02T03:08:26.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1095213056 bytes 2023-12-02T03:08:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1134353408 bytes 2023-12-02T03:08:26.884 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~16s 2023-12-02T03:08:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1210820608 bytes 2023-12-02T03:08:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1261553664 bytes 2023-12-02T03:08:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1350680576 bytes 2023-12-02T03:08:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1376737280 bytes 2023-12-02T03:08:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1399449600 bytes 2023-12-02T03:08:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1426030592 bytes 2023-12-02T03:08:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1446829056 bytes 2023-12-02T03:08:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1512827904 bytes 2023-12-02T03:08:27.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1580773376 bytes 2023-12-02T03:08:27.854 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1609883648 bytes 2023-12-02T03:08:27.940 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1639840768 bytes 2023-12-02T03:08:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1686116352 bytes 2023-12-02T03:08:28.149 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1720289280 bytes 2023-12-02T03:08:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1771991040 bytes 2023-12-02T03:08:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1820187648 bytes 2023-12-02T03:08:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1895335936 bytes 2023-12-02T03:08:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1936624640 bytes 2023-12-02T03:08:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1954425856 bytes 2023-12-02T03:08:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 2006593536 bytes 2023-12-02T03:08:28.883 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 2085606400 bytes 2023-12-02T03:08:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2141038592 bytes 2023-12-02T03:08:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2183037952 bytes 2023-12-02T03:08:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2222243840 bytes 2023-12-02T03:08:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2312187904 bytes 2023-12-02T03:08:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2370741248 bytes 2023-12-02T03:08:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2429324288 bytes 2023-12-02T03:08:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2453407744 bytes 2023-12-02T03:08:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2474318848 bytes 2023-12-02T03:08:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2534499328 bytes 2023-12-02T03:08:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2540478464 bytes 2023-12-02T03:08:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2563974144 bytes 2023-12-02T03:08:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2593179648 bytes 2023-12-02T03:08:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2620838912 bytes 2023-12-02T03:08:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2733935616 bytes 2023-12-02T03:08:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2734560256 bytes 2023-12-02T03:08:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2777329664 bytes 2023-12-02T03:08:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2779553792 bytes 2023-12-02T03:08:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2817376256 bytes 2023-12-02T03:08:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2870337536 bytes 2023-12-02T03:08:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2904512512 bytes 2023-12-02T03:08:30.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2993262592 bytes 2023-12-02T03:08:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 3031540736 bytes 2023-12-02T03:08:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 3079861248 bytes 2023-12-02T03:08:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 3125315584 bytes 2023-12-02T03:08:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 3176543232 bytes 2023-12-02T03:08:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 3210807296 bytes 2023-12-02T03:08:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3254061056 bytes 2023-12-02T03:08:31.563 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3287216128 bytes 2023-12-02T03:08:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3318836224 bytes 2023-12-02T03:08:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3398574080 bytes 2023-12-02T03:08:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3454167040 bytes 2023-12-02T03:08:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3456485376 bytes 2023-12-02T03:08:32.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3527641088 bytes 2023-12-02T03:08:32.146 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7464226370132041) > weight (0.25) 2023-12-02T03:08:32.146 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9734200462116983) > weight (0.25) 2023-12-02T03:08:32.146 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-02T03:08:32.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-02T03:08:32.147 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-02T03:08:32.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi057.stderr:2023-12-02T03:08:32.145+0000 7f374bb9e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 100375) UID: 1000 2023-12-02T03:08:32.187 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3627097088 bytes 2023-12-02T03:08:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:08:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:08:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:08:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:08:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:08:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_48 2023-12-02T03:08:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2023-12-02T03:08:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:08:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:08:32.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:32.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_48 2023-12-02T03:08:32.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:32.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:08:32.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:08:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_48 128M 2023-12-02T03:08:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:08:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:08:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_48 2023-12-02T03:08:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:08:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:08:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:08:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_48 2023-12-02T03:08:32.287 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~22s 2023-12-02T03:08:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:08:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:08:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:08:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:08:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2023-12-02T03:08:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:08:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:08:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_48 2023-12-02T03:08:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:08:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2023-12-02T03:08:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:08:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2023-12-02T03:08:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_48': 2023-12-02T03:08:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:08:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:08:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:08:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 1491cfb00be8 2023-12-02T03:08:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.1491cfb00be8 2023-12-02T03:08:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:08:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:08:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:08:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:08:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:08:32 2023 2023-12-02T03:08:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:08:32 2023 2023-12-02T03:08:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:08:32 2023 2023-12-02T03:08:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:08:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:08:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 4f77376e-57d2-4b09-b376-6129ff7390b9 2023-12-02T03:08:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:08:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:08:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_48 2023-12-02T03:08:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:08:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:08:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_48 2023-12-02T03:08:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:08:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=5 2023-12-02T03:08:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:08:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:08:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 46508032 bytes 2023-12-02T03:08:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 101763072 bytes 2023-12-02T03:08:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 152763392 bytes 2023-12-02T03:08:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 208952320 bytes 2023-12-02T03:08:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 233975808 bytes 2023-12-02T03:08:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 295101440 bytes 2023-12-02T03:08:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 338882560 bytes 2023-12-02T03:08:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 390502400 bytes 2023-12-02T03:08:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 461366272 bytes 2023-12-02T03:08:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 528790528 bytes 2023-12-02T03:08:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 573222912 bytes 2023-12-02T03:08:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 591661056 bytes 2023-12-02T03:08:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 669864960 bytes 2023-12-02T03:08:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 716971008 bytes 2023-12-02T03:08:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 867166208 bytes 2023-12-02T03:08:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 922859520 bytes 2023-12-02T03:08:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 940829696 bytes 2023-12-02T03:08:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 950420480 bytes 2023-12-02T03:08:35.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 1000077312 bytes 2023-12-02T03:08:35.358 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 1041810432 bytes 2023-12-02T03:08:35.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 1081447424 bytes 2023-12-02T03:08:35.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 1082085376 bytes 2023-12-02T03:08:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 1095812096 bytes 2023-12-02T03:08:35.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1110777856 bytes 2023-12-02T03:08:35.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1133952000 bytes 2023-12-02T03:08:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1193568256 bytes 2023-12-02T03:08:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1246036992 bytes 2023-12-02T03:08:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1279406080 bytes 2023-12-02T03:08:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1333195776 bytes 2023-12-02T03:08:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1350768640 bytes 2023-12-02T03:08:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1396928512 bytes 2023-12-02T03:08:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1424707584 bytes 2023-12-02T03:08:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1524444160 bytes 2023-12-02T03:08:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1562302464 bytes 2023-12-02T03:08:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1608384512 bytes 2023-12-02T03:08:36.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1675905024 bytes 2023-12-02T03:08:36.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1693360128 bytes 2023-12-02T03:08:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1700059136 bytes 2023-12-02T03:08:37.106 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1758971904 bytes 2023-12-02T03:08:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1803671552 bytes 2023-12-02T03:08:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1815022592 bytes 2023-12-02T03:08:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1881530368 bytes 2023-12-02T03:08:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1965991936 bytes 2023-12-02T03:08:37.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1985179648 bytes 2023-12-02T03:08:37.692 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~27s 2023-12-02T03:08:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 2049389568 bytes 2023-12-02T03:08:37.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 2094696448 bytes 2023-12-02T03:08:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 2173119488 bytes 2023-12-02T03:08:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2212355072 bytes 2023-12-02T03:08:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2220559360 bytes 2023-12-02T03:08:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2279781376 bytes 2023-12-02T03:08:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:08:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:08:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:08:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:08:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:08:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_49 2023-12-02T03:08:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2023-12-02T03:08:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:08:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:08:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_49 2023-12-02T03:08:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:08:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:08:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_49 128M 2023-12-02T03:08:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:08:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:08:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_49 2023-12-02T03:08:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:08:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:08:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:08:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_49 2023-12-02T03:08:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:08:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:08:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:08:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:08:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2023-12-02T03:08:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:08:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:08:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_49 2023-12-02T03:08:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:08:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2023-12-02T03:08:39.038 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8644007120653207) > weight (0.25) 2023-12-02T03:08:39.039 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8064925014700463) > weight (0.25) 2023-12-02T03:08:39.039 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5409866646210366) > weight (0.25) 2023-12-02T03:08:39.039 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9960728639496856) > weight (0.25) 2023-12-02T03:08:39.039 INFO:tasks.rbd_mirror_thrash:waiting for 65 secs before thrashing 2023-12-02T03:08:39.148 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-02T03:08:39.148 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T03:08:39.148 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-02T03:08:39.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-02T03:08:39.149 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-02T03:08:39.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-02T03:08:39.152 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2023-12-02T03:08:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:08:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2023-12-02T03:08:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_49': 2023-12-02T03:08:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:08:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:08:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:08:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 149aa0970a9a 2023-12-02T03:08:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.149aa0970a9a 2023-12-02T03:08:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:08:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:08:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:08:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:08:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:08:38 2023 2023-12-02T03:08:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:08:38 2023 2023-12-02T03:08:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:08:38 2023 2023-12-02T03:08:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:08:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:08:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 92a00385-f1d8-4e86-abb7-0e1810b799c3 2023-12-02T03:08:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:08:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:08:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_49 2023-12-02T03:08:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:08:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:08:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_49 2023-12-02T03:08:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:08:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=14 2023-12-02T03:08:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:08:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:08:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 21171200 bytes 2023-12-02T03:08:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 57507840 bytes 2023-12-02T03:08:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 123752448 bytes 2023-12-02T03:08:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 147977216 bytes 2023-12-02T03:08:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 166470656 bytes 2023-12-02T03:08:39.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 241397760 bytes 2023-12-02T03:08:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 254155776 bytes 2023-12-02T03:08:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 327281664 bytes 2023-12-02T03:08:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 385272832 bytes 2023-12-02T03:08:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 409351168 bytes 2023-12-02T03:08:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 500268032 bytes 2023-12-02T03:08:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 526761984 bytes 2023-12-02T03:08:40.660 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 530230272 bytes 2023-12-02T03:08:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 532122624 bytes 2023-12-02T03:08:40.822 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 580589568 bytes 2023-12-02T03:08:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 610478080 bytes 2023-12-02T03:08:41.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 631322624 bytes 2023-12-02T03:08:41.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 679512064 bytes 2023-12-02T03:08:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 721720320 bytes 2023-12-02T03:08:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 813692928 bytes 2023-12-02T03:08:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 860717056 bytes 2023-12-02T03:08:41.522 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 869244928 bytes 2023-12-02T03:08:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 922547200 bytes 2023-12-02T03:08:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1019379712 bytes 2023-12-02T03:08:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1042723840 bytes 2023-12-02T03:08:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1074651136 bytes 2023-12-02T03:08:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1168851968 bytes 2023-12-02T03:08:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1213065216 bytes 2023-12-02T03:08:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1259882496 bytes 2023-12-02T03:08:42.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1291119616 bytes 2023-12-02T03:08:42.484 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1333052416 bytes 2023-12-02T03:08:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1354191872 bytes 2023-12-02T03:08:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1431794688 bytes 2023-12-02T03:08:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1451980800 bytes 2023-12-02T03:08:42.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1483929600 bytes 2023-12-02T03:08:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1530789888 bytes 2023-12-02T03:08:43.095 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~32s 2023-12-02T03:08:43.098 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1581852672 bytes 2023-12-02T03:08:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1621816320 bytes 2023-12-02T03:08:43.281 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1645242368 bytes 2023-12-02T03:08:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1678446592 bytes 2023-12-02T03:08:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1707785216 bytes 2023-12-02T03:08:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1725279232 bytes 2023-12-02T03:08:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1751292928 bytes 2023-12-02T03:08:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1823741952 bytes 2023-12-02T03:08:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1899803648 bytes 2023-12-02T03:08:44.037 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1972247552 bytes 2023-12-02T03:08:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 1996773376 bytes 2023-12-02T03:08:44.184 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2001545216 bytes 2023-12-02T03:08:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2071773184 bytes 2023-12-02T03:08:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2093517824 bytes 2023-12-02T03:08:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2163996672 bytes 2023-12-02T03:08:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2222153728 bytes 2023-12-02T03:08:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2280488960 bytes 2023-12-02T03:08:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2334914560 bytes 2023-12-02T03:08:44.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2393157632 bytes 2023-12-02T03:08:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2454635520 bytes 2023-12-02T03:08:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2465006592 bytes 2023-12-02T03:08:45.233 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2487291904 bytes 2023-12-02T03:08:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2538611712 bytes 2023-12-02T03:08:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2579060736 bytes 2023-12-02T03:08:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6100 IOs, 2610937856 bytes 2023-12-02T03:08:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6200 IOs, 2621131776 bytes 2023-12-02T03:08:45.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6300 IOs, 2633705472 bytes 2023-12-02T03:08:45.782 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6400 IOs, 2670158848 bytes 2023-12-02T03:08:45.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6500 IOs, 2718978048 bytes 2023-12-02T03:08:46.005 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6600 IOs, 2776082432 bytes 2023-12-02T03:08:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6700 IOs, 2871741440 bytes 2023-12-02T03:08:46.242 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6800 IOs, 2888625152 bytes 2023-12-02T03:08:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6900 IOs, 2917722112 bytes 2023-12-02T03:08:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7000 IOs, 2918345728 bytes 2023-12-02T03:08:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7100 IOs, 2983049216 bytes 2023-12-02T03:08:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7200 IOs, 3031482368 bytes 2023-12-02T03:08:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7300 IOs, 3041965056 bytes 2023-12-02T03:08:46.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7400 IOs, 3059926016 bytes 2023-12-02T03:08:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7500 IOs, 3102033920 bytes 2023-12-02T03:08:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7600 IOs, 3182016512 bytes 2023-12-02T03:08:47.142 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7700 IOs, 3276044288 bytes 2023-12-02T03:08:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7800 IOs, 3355526144 bytes 2023-12-02T03:08:47.346 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:7900 IOs, 3357918208 bytes 2023-12-02T03:08:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8000 IOs, 3443406848 bytes 2023-12-02T03:08:47.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8100 IOs, 3467642880 bytes 2023-12-02T03:08:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8200 IOs, 3510810624 bytes 2023-12-02T03:08:47.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8300 IOs, 3554648064 bytes 2023-12-02T03:08:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8400 IOs, 3612526592 bytes 2023-12-02T03:08:47.962 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8500 IOs, 3632808960 bytes 2023-12-02T03:08:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8600 IOs, 3671795712 bytes 2023-12-02T03:08:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8700 IOs, 3743544320 bytes 2023-12-02T03:08:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8800 IOs, 3788497920 bytes 2023-12-02T03:08:48.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:8900 IOs, 3815062528 bytes 2023-12-02T03:08:48.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~38s 2023-12-02T03:08:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9000 IOs, 3861941248 bytes 2023-12-02T03:08:48.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9100 IOs, 3893003264 bytes 2023-12-02T03:08:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9200 IOs, 3976620032 bytes 2023-12-02T03:08:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9300 IOs, 3991458816 bytes 2023-12-02T03:08:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9400 IOs, 4003096576 bytes 2023-12-02T03:08:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9500 IOs, 4026301440 bytes 2023-12-02T03:08:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9600 IOs, 4079889408 bytes 2023-12-02T03:08:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9700 IOs, 4120776704 bytes 2023-12-02T03:08:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9800 IOs, 4167884800 bytes 2023-12-02T03:08:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:9900 IOs, 4212608000 bytes 2023-12-02T03:08:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10000 IOs, 4288323584 bytes 2023-12-02T03:08:49.638 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10100 IOs, 4356890624 bytes 2023-12-02T03:08:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10200 IOs, 4384840704 bytes 2023-12-02T03:08:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10300 IOs, 4404683776 bytes 2023-12-02T03:08:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10400 IOs, 4436822016 bytes 2023-12-02T03:08:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10500 IOs, 4468674560 bytes 2023-12-02T03:08:50.059 INFO:tasks.ceph.cluster2.osd.2.smithi057.stderr:2023-12-02T03:08:50.057+0000 7f34767b9640 -1 reset not still connected to 0x562263507110 2023-12-02T03:08:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10600 IOs, 4490565632 bytes 2023-12-02T03:08:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10700 IOs, 4504715264 bytes 2023-12-02T03:08:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10800 IOs, 4579542016 bytes 2023-12-02T03:08:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:10900 IOs, 4605618176 bytes 2023-12-02T03:08:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11000 IOs, 4696140800 bytes 2023-12-02T03:08:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11100 IOs, 4699934720 bytes 2023-12-02T03:08:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11200 IOs, 4727047168 bytes 2023-12-02T03:08:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11300 IOs, 4755698688 bytes 2023-12-02T03:08:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11400 IOs, 4778664960 bytes 2023-12-02T03:08:50.940 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11500 IOs, 4810735616 bytes 2023-12-02T03:08:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11600 IOs, 4827965440 bytes 2023-12-02T03:08:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11700 IOs, 4853243904 bytes 2023-12-02T03:08:51.235 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11800 IOs, 4933194752 bytes 2023-12-02T03:08:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:11900 IOs, 4933825536 bytes 2023-12-02T03:08:51.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:12000 IOs, 4976833536 bytes 2023-12-02T03:08:51.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:12100 IOs, 5007669248 bytes 2023-12-02T03:08:51.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:12200 IOs, 5022615552 bytes 2023-12-02T03:08:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:12300 IOs, 5053872128 bytes 2023-12-02T03:08:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:12400 IOs, 5086769152 bytes 2023-12-02T03:08:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:12500 IOs, 5143580672 bytes 2023-12-02T03:08:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:12600 IOs, 5195147264 bytes 2023-12-02T03:08:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:12700 IOs, 5257797632 bytes 2023-12-02T03:08:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:12800 IOs, 5258414080 bytes 2023-12-02T03:08:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:12900 IOs, 5319811072 bytes 2023-12-02T03:08:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:13000 IOs, 5363027968 bytes 2023-12-02T03:08:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:13100 IOs, 5445581824 bytes 2023-12-02T03:08:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:13200 IOs, 5483682816 bytes 2023-12-02T03:08:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:13300 IOs, 5498485760 bytes 2023-12-02T03:08:52.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:13400 IOs, 5562866688 bytes 2023-12-02T03:08:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:13500 IOs, 5594345472 bytes 2023-12-02T03:08:53.021 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:13600 IOs, 5627991040 bytes 2023-12-02T03:08:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:13700 IOs, 5686010880 bytes 2023-12-02T03:08:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:13800 IOs, 5720899584 bytes 2023-12-02T03:08:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:13900 IOs, 5763828736 bytes 2023-12-02T03:08:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:08:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:08:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:08:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:08:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:08:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_50 2023-12-02T03:08:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2023-12-02T03:08:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:08:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:08:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_50 2023-12-02T03:08:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:08:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n snapshot ']' 2023-12-02T03:08:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_image cluster2 mirror image_50 128M 2023-12-02T03:08:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:08:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:08:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_50 2023-12-02T03:08:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local size=128 2023-12-02T03:08:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n 128M ']' 2023-12-02T03:08:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ size=128M 2023-12-02T03:08:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ shift 2023-12-02T03:08:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_50 2023-12-02T03:08:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ is_pool_mirror_mode_image mirror 2023-12-02T03:08:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:08:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' image = image ']' 2023-12-02T03:08:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:08:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2023-12-02T03:08:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:08:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:08:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_50 2023-12-02T03:08:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local mode=snapshot 2023-12-02T03:08:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2023-12-02T03:08:53.901 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~43s 2023-12-02T03:08:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirroring enabled 2023-12-02T03:08:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2023-12-02T03:08:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_50': 2023-12-02T03:08:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:08:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:08:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:08:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 14af86f6347d 2023-12-02T03:08:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.14af86f6347d 2023-12-02T03:08:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:08:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering 2023-12-02T03:08:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:08:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:08:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:08:53 2023 2023-12-02T03:08:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:08:53 2023 2023-12-02T03:08:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:08:53 2023 2023-12-02T03:08:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:08:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:08:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: f939d359-e3e3-4881-b4c7-c05064e62969 2023-12-02T03:08:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: true 2023-12-02T03:08:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -n '' ']' 2023-12-02T03:08:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stress_write_image cluster2 mirror image_50 2023-12-02T03:08:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:08:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:08:54.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_50 2023-12-02T03:08:54.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-02T03:08:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local duration=6 2023-12-02T03:08:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:08:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2023-12-02T03:08:54.859 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:100 IOs, 47344640 bytes 2023-12-02T03:08:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:200 IOs, 116489216 bytes 2023-12-02T03:08:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:300 IOs, 144253952 bytes 2023-12-02T03:08:55.098 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:400 IOs, 147355648 bytes 2023-12-02T03:08:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:500 IOs, 175404032 bytes 2023-12-02T03:08:55.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:600 IOs, 242302976 bytes 2023-12-02T03:08:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:700 IOs, 307888128 bytes 2023-12-02T03:08:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:800 IOs, 371160064 bytes 2023-12-02T03:08:55.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:900 IOs, 398129152 bytes 2023-12-02T03:08:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1000 IOs, 436924416 bytes 2023-12-02T03:08:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1100 IOs, 488641536 bytes 2023-12-02T03:08:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1200 IOs, 538724352 bytes 2023-12-02T03:08:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1300 IOs, 549908480 bytes 2023-12-02T03:08:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1400 IOs, 579647488 bytes 2023-12-02T03:08:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1500 IOs, 580264960 bytes 2023-12-02T03:08:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1600 IOs, 645552128 bytes 2023-12-02T03:08:56.328 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1700 IOs, 721172480 bytes 2023-12-02T03:08:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1800 IOs, 793665536 bytes 2023-12-02T03:08:56.530 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:1900 IOs, 841778176 bytes 2023-12-02T03:08:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2000 IOs, 897193984 bytes 2023-12-02T03:08:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2100 IOs, 954198016 bytes 2023-12-02T03:08:56.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2200 IOs, 1030998016 bytes 2023-12-02T03:08:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2300 IOs, 1100753920 bytes 2023-12-02T03:08:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2400 IOs, 1119769600 bytes 2023-12-02T03:08:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2500 IOs, 1140070400 bytes 2023-12-02T03:08:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2600 IOs, 1141935104 bytes 2023-12-02T03:08:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2700 IOs, 1196129280 bytes 2023-12-02T03:08:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2800 IOs, 1221511168 bytes 2023-12-02T03:08:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:2900 IOs, 1259976704 bytes 2023-12-02T03:08:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3000 IOs, 1309632512 bytes 2023-12-02T03:08:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3100 IOs, 1404900352 bytes 2023-12-02T03:08:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3200 IOs, 1421604864 bytes 2023-12-02T03:08:57.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3300 IOs, 1424760832 bytes 2023-12-02T03:08:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3400 IOs, 1482487808 bytes 2023-12-02T03:08:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3500 IOs, 1542715392 bytes 2023-12-02T03:08:58.191 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3600 IOs, 1545425920 bytes 2023-12-02T03:08:58.283 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3700 IOs, 1586018304 bytes 2023-12-02T03:08:58.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3800 IOs, 1668701184 bytes 2023-12-02T03:08:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:3900 IOs, 1716250624 bytes 2023-12-02T03:08:58.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4000 IOs, 1749406720 bytes 2023-12-02T03:08:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4100 IOs, 1757329408 bytes 2023-12-02T03:08:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4200 IOs, 1804441600 bytes 2023-12-02T03:08:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4300 IOs, 1882809344 bytes 2023-12-02T03:08:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4400 IOs, 1906619392 bytes 2023-12-02T03:08:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4500 IOs, 1907252224 bytes 2023-12-02T03:08:59.143 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4600 IOs, 1996368896 bytes 2023-12-02T03:08:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4700 IOs, 2058347520 bytes 2023-12-02T03:08:59.305 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~49s 2023-12-02T03:08:59.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4800 IOs, 2099651584 bytes 2023-12-02T03:08:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:4900 IOs, 2138785792 bytes 2023-12-02T03:08:59.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5000 IOs, 2179086336 bytes 2023-12-02T03:08:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5100 IOs, 2210902016 bytes 2023-12-02T03:08:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5200 IOs, 2257089536 bytes 2023-12-02T03:08:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5300 IOs, 2294280192 bytes 2023-12-02T03:08:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5400 IOs, 2323470336 bytes 2023-12-02T03:08:59.991 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5500 IOs, 2345447424 bytes 2023-12-02T03:09:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5600 IOs, 2371425280 bytes 2023-12-02T03:09:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5700 IOs, 2414405632 bytes 2023-12-02T03:09:00.274 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5800 IOs, 2455918592 bytes 2023-12-02T03:09:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:5900 IOs, 2508674048 bytes 2023-12-02T03:09:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:6000 IOs, 2560442368 bytes 2023-12-02T03:09:00.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ error_code=124 2023-12-02T03:09:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set -e 2023-12-02T03:09:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 124 -eq 124 ']' 2023-12-02T03:09:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:09:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_pool_images cluster2 mirror 50 2023-12-02T03:09:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:09:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image_count=50 2023-12-02T03:09:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:09:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local count 2023-12-02T03:09:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local last_count=0 2023-12-02T03:09:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:09:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ seq 1 40 2023-12-02T03:09:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in `seq 1 40` 2023-12-02T03:09:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 1 -ne 1 2023-12-02T03:09:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2023-12-02T03:09:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep 'images: ' 2023-12-02T03:09:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cut '-d ' -f 2 2023-12-02T03:09:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T03:09:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ count=50 2023-12-02T03:09:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 50 = 50 2023-12-02T03:09:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:09:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_pool_healthy cluster2 mirror 2023-12-02T03:09:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:09:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:09:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state 2023-12-02T03:09:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ seq 1 40 2023-12-02T03:09:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in `seq 1 40` 2023-12-02T03:09:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 1 -ne 1 2023-12-02T03:09:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2023-12-02T03:09:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep 'image health:' 2023-12-02T03:09:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cut '-d ' -f 3 2023-12-02T03:09:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T03:09:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ state=OK 2023-12-02T03:09:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test OK = ERROR 2023-12-02T03:09:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test OK = OK 2023-12-02T03:09:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:09:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_pool_images cluster1 mirror 50 2023-12-02T03:09:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:09:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image_count=50 2023-12-02T03:09:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:09:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local count 2023-12-02T03:09:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local last_count=0 2023-12-02T03:09:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:09:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ seq 1 40 2023-12-02T03:09:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in `seq 1 40` 2023-12-02T03:09:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 1 -ne 1 2023-12-02T03:09:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-02T03:09:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep 'images: ' 2023-12-02T03:09:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cut '-d ' -f 2 2023-12-02T03:09:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T03:09:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ count=50 2023-12-02T03:09:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 50 = 50 2023-12-02T03:09:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:09:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_pool_healthy cluster1 mirror 2023-12-02T03:09:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:09:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:09:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state 2023-12-02T03:09:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ seq 1 40 2023-12-02T03:09:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in `seq 1 40` 2023-12-02T03:09:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 1 -ne 1 2023-12-02T03:09:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-02T03:09:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep 'image health:' 2023-12-02T03:09:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cut '-d ' -f 3 2023-12-02T03:09:00.823 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T03:09:00.826 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ state=WARNING 2023-12-02T03:09:00.826 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test WARNING = ERROR 2023-12-02T03:09:00.826 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test WARNING = OK 2023-12-02T03:09:00.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in `seq 1 40` 2023-12-02T03:09:00.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 2 -ne 1 2023-12-02T03:09:00.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 30 2023-12-02T03:09:04.709 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~54s 2023-12-02T03:09:10.113 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~59s 2023-12-02T03:09:15.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~65s 2023-12-02T03:09:20.920 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~70s 2023-12-02T03:09:26.324 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~76s 2023-12-02T03:09:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-02T03:09:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep 'image health:' 2023-12-02T03:09:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cut '-d ' -f 3 2023-12-02T03:09:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: failed to query services: (13) Permission denied 2023-12-02T03:09:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ state=OK 2023-12-02T03:09:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test OK = ERROR 2023-12-02T03:09:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test OK = OK 2023-12-02T03:09:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:09:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ testlog 'TEST: compare many images' 2023-12-02T03:09:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ tee -a /tmp/tmp.kebs98RNzE/rbd-mirror.test.log 2023-12-02T03:09:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ date '+%F %T' 2023-12-02T03:09:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02 03:09:30 TEST: compare many images 2023-12-02T03:09:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:2023-12-02 03:09:30 TEST: compare many images 2023-12-02T03:09:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ seq 1 50 2023-12-02T03:09:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:09:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_1 2023-12-02T03:09:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror image_1 snap 2023-12-02T03:09:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:09:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_1 2023-12-02T03:09:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:09:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T03:09:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2023-12-02T03:09:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:09:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_1 2023-12-02T03:09:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2023-12-02T03:09:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:09:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_1 2023-12-02T03:09:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T03:09:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:09:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:09:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:09:31.728 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~81s 2023-12-02T03:09:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2023-12-02T03:09:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:09:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_1 2023-12-02T03:09:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T03:09:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T03:09:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T03:09:32.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2023-12-02T03:09:32.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T03:09:32.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T03:09:32.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:09:32.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T03:09:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T03:09:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T03:09:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:09:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T03:09:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:09:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:09:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:09:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2023-12-02T03:09:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:09:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:09:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:09:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:09:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:09:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:09:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:09:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:09:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:09:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:09:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:09:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:09:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:09:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:09:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:09:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:09:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:09:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:09:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:09:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:09:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:09:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:09:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:09:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:09:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:09:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:09:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:09:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:09:32.698 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:09:32.699 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:09:32.699 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:09:32.699 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2023-12-02T03:09:33.025 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T03:09:33.026 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T03:09:33.026 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T03:09:33.026 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T03:09:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T03:09:33.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T03:09:33.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T03:09:33.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:09:33.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2023-12-02T03:09:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:09:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:09:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_1 2023-12-02T03:09:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T03:09:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T03:09:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2023-12-02T03:09:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:09:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:09:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_1 2023-12-02T03:09:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-image_1.status 2023-12-02T03:09:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-image_1.status 2023-12-02T03:09:33.030 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:09:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_1.status 2023-12-02T03:09:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-image_1.status 2023-12-02T03:09:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-image_1.status 2023-12-02T03:09:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:09:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_1.status 2023-12-02T03:09:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2023-12-02T03:09:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:09:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_1 2023-12-02T03:09:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2023-12-02T03:09:33.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 119 2023-12-02T03:09:33.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.kebs98RNzE/cluster2-mirror-image_1.status 2023-12-02T03:09:33.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:09:33.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:33.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_1 2023-12-02T03:09:33.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_1.status 2023-12-02T03:09:33.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2023-12-02T03:09:33.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:09:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T03:09:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=119 2023-12-02T03:09:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:09:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:09:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T03:09:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_1.status 2023-12-02T03:09:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:09:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_1 2023-12-02T03:09:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_1.status 2023-12-02T03:09:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-02T03:09:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:09:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:09:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=68 2023-12-02T03:09:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 119 = 68 2023-12-02T03:09:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:09:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T03:09:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_1.status 2023-12-02T03:09:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:09:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_1 2023-12-02T03:09:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_1.status 2023-12-02T03:09:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-02T03:09:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:09:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:09:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=68 2023-12-02T03:09:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 119 = 68 2023-12-02T03:09:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:09:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T03:09:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_1.status 2023-12-02T03:09:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:09:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_1 2023-12-02T03:09:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_1.status 2023-12-02T03:09:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-02T03:09:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:09:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:09:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:09:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:09:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:09:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:09:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:09:35.315 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 119 = '' 2023-12-02T03:09:35.315 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:09:35.315 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1.6 2023-12-02T03:09:36.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_1.status 2023-12-02T03:09:36.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:09:36.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:36.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_1 2023-12-02T03:09:36.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_1.status 2023-12-02T03:09:36.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-02T03:09:36.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:09:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:09:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=119 2023-12-02T03:09:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 119 = 119 2023-12-02T03:09:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:09:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2023-12-02T03:09:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:09:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_1 2023-12-02T03:09:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:09:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:09:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T03:09:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:09:37.132 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~86s 2023-12-02T03:09:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2023-12-02T03:09:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_1': 2023-12-02T03:09:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:09:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:09:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:09:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 10e4cf0fd5b2 2023-12-02T03:09:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.10e4cf0fd5b2 2023-12-02T03:09:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:09:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:09:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:09:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:09:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 02:59:41 2023 2023-12-02T03:09:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 02:59:41 2023 2023-12-02T03:09:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 02:59:41 2023 2023-12-02T03:09:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T03:09:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:09:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:09:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 49c16484-3472-4671-8be8-03c1846564a3 2023-12-02T03:09:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:09:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:09:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror image_1 snap 2023-12-02T03:09:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_1 2023-12-02T03:09:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:09:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T03:09:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_1.export 2023-12-02T03:09:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_1.export 2023-12-02T03:09:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_1.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_1.export 2023-12-02T03:09:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.kebs98RNzE/cluster2-mirror-image_1.export 2023-12-02T03:09:38.226 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:09:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.kebs98RNzE/cluster1-mirror-image_1.export 2023-12-02T03:09:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:09:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-image_1.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_1.export 2023-12-02T03:09:38.570 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_1.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_1.export 2023-12-02T03:09:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:09:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:09:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_2 2023-12-02T03:09:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror image_2 snap 2023-12-02T03:09:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:09:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_2 2023-12-02T03:09:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:09:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T03:09:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2023-12-02T03:09:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:09:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_2 2023-12-02T03:09:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2023-12-02T03:09:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:09:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_2 2023-12-02T03:09:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T03:09:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:09:39.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:09:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:09:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2023-12-02T03:09:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:09:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_2 2023-12-02T03:09:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T03:09:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T03:09:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T03:09:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2023-12-02T03:09:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T03:09:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T03:09:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:09:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T03:09:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T03:09:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T03:09:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:09:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T03:09:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:09:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:09:40.289 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:09:40.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2023-12-02T03:09:40.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:09:40.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:09:40.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:09:40.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:09:40.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:09:40.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:09:40.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:09:40.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:09:40.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:09:40.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:09:40.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:09:40.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:09:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:09:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:09:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:09:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:09:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:09:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:09:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:09:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:09:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:09:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:09:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:09:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:09:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:09:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:09:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:09:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:09:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:09:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:09:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:09:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2023-12-02T03:09:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:no valid command found; 3 closest matches: 2023-12-02T03:09:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_24 2023-12-02T03:09:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_26 2023-12-02T03:09:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_28 2023-12-02T03:09:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:09:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:09:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2023-12-02T03:09:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:09:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:09:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:09:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:09:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:09:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:09:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:09:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:09:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:09:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:09:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:09:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:09:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:09:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:09:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:09:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:09:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:09:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:09:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:09:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:09:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:09:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:09:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:09:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:09:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:09:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:09:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:09:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:09:40.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:09:40.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:09:40.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:09:40.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2023-12-02T03:09:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T03:09:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T03:09:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T03:09:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T03:09:40.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T03:09:40.523 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T03:09:40.523 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T03:09:40.523 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:09:40.523 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2023-12-02T03:09:40.523 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:09:40.523 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:09:40.523 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:40.523 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_2 2023-12-02T03:09:40.523 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T03:09:40.523 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T03:09:40.523 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2023-12-02T03:09:40.523 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:09:40.523 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:09:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_2 2023-12-02T03:09:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-image_2.status 2023-12-02T03:09:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-image_2.status 2023-12-02T03:09:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:09:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_2.status 2023-12-02T03:09:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-image_2.status 2023-12-02T03:09:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-image_2.status 2023-12-02T03:09:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:09:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_2.status 2023-12-02T03:09:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2023-12-02T03:09:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:09:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_2 2023-12-02T03:09:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2023-12-02T03:09:41.307 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 121 2023-12-02T03:09:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.kebs98RNzE/cluster2-mirror-image_2.status 2023-12-02T03:09:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:09:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_2 2023-12-02T03:09:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_2.status 2023-12-02T03:09:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2023-12-02T03:09:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:09:41.362 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T03:09:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=121 2023-12-02T03:09:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:09:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:09:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T03:09:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_2.status 2023-12-02T03:09:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:09:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_2 2023-12-02T03:09:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_2.status 2023-12-02T03:09:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-02T03:09:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:09:41.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:09:41.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=69 2023-12-02T03:09:41.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 121 = 69 2023-12-02T03:09:41.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:09:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T03:09:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_2.status 2023-12-02T03:09:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:09:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_2 2023-12-02T03:09:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_2.status 2023-12-02T03:09:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-02T03:09:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:09:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:09:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:09:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:09:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:09:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:09:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:09:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 121 = '' 2023-12-02T03:09:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:09:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T03:09:42.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~92s 2023-12-02T03:09:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_2.status 2023-12-02T03:09:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:09:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_2 2023-12-02T03:09:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_2.status 2023-12-02T03:09:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-02T03:09:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:09:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:09:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=121 2023-12-02T03:09:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 121 = 121 2023-12-02T03:09:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:09:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2023-12-02T03:09:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:09:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_2 2023-12-02T03:09:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:09:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:09:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T03:09:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:09:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2023-12-02T03:09:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_2': 2023-12-02T03:09:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:09:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:09:43.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:09:43.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 115ef4dd5c1d 2023-12-02T03:09:43.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.115ef4dd5c1d 2023-12-02T03:09:43.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:09:43.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:09:43.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:09:43.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:09:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 02:59:54 2023 2023-12-02T03:09:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 02:59:54 2023 2023-12-02T03:09:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 02:59:54 2023 2023-12-02T03:09:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T03:09:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:09:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:09:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 09e06fe9-a175-4009-952b-403e63e1ac10 2023-12-02T03:09:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:09:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:09:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror image_2 snap 2023-12-02T03:09:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_2 2023-12-02T03:09:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:09:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T03:09:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_2.export 2023-12-02T03:09:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_2.export 2023-12-02T03:09:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_2.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_2.export 2023-12-02T03:09:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.kebs98RNzE/cluster2-mirror-image_2.export 2023-12-02T03:09:44.040 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-02T03:09:44.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-02T03:09:44.040 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-02T03:09:44.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi057.stderr:2023-12-02T03:09:44.039+0000 7f7015aed640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 100570) UID: 1000 2023-12-02T03:09:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:09:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.kebs98RNzE/cluster1-mirror-image_2.export 2023-12-02T03:09:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:09:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-image_2.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_2.export 2023-12-02T03:09:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_2.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_2.export 2023-12-02T03:09:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:09:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:09:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_3 2023-12-02T03:09:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror image_3 snap 2023-12-02T03:09:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:09:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_3 2023-12-02T03:09:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:09:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T03:09:45.153 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6652157577891741) > weight (0.25) 2023-12-02T03:09:45.153 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-02T03:09:45.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-02T03:09:45.153 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-02T03:09:45.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi057.stderr:2023-12-02T03:09:45.151+0000 7fb80640b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 96283) UID: 1000 2023-12-02T03:09:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2023-12-02T03:09:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:09:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_3 2023-12-02T03:09:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2023-12-02T03:09:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:09:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_3 2023-12-02T03:09:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T03:09:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:09:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:09:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:09:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2023-12-02T03:09:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:09:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_3 2023-12-02T03:09:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T03:09:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T03:09:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T03:09:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T03:09:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2023-12-02T03:09:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T03:09:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:09:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T03:09:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T03:09:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T03:09:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:09:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T03:09:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:09:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:09:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:09:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2023-12-02T03:09:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:09:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:09:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:09:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:09:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:09:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:09:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:09:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:09:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:09:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:09:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:09:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:09:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:09:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:09:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:09:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:09:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:09:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:09:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:09:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:09:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:09:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:09:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:09:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:09:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:09:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:09:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:09:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:09:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:09:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:09:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:09:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2023-12-02T03:09:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-02T03:09:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:09:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2023-12-02T03:09:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:09:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:09:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:09:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:09:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:09:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:09:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:09:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:09:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:09:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:09:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:09:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:09:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:09:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:09:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:09:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:09:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:09:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:09:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:09:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:09:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:09:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:09:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:09:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:09:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:09:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:09:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:09:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:09:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:09:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:09:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:09:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2023-12-02T03:09:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:no valid command found; 3 closest matches: 2023-12-02T03:09:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_31 2023-12-02T03:09:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_35 2023-12-02T03:09:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_39 2023-12-02T03:09:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:09:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:09:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_3 2023-12-02T03:09:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T03:09:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:09:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:09:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T03:09:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T03:09:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:09:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:09:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:09:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T03:09:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T03:09:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:09:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T03:09:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:09:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T03:09:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T03:09:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:09:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:09:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T03:09:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T03:09:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:09:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:09:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:09:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T03:09:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:09:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:09:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T03:09:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T03:09:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T03:09:46.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:09:46.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:09:46.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:09:46.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_3 2023-12-02T03:09:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:no valid command found; 3 closest matches: 2023-12-02T03:09:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_32 2023-12-02T03:09:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_33 2023-12-02T03:09:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_37 2023-12-02T03:09:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:09:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:09:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_3 2023-12-02T03:09:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T03:09:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:09:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:09:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T03:09:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T03:09:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:09:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:09:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:09:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T03:09:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T03:09:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:09:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T03:09:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:09:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T03:09:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T03:09:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:09:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:09:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T03:09:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T03:09:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:09:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:09:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:09:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T03:09:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:09:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:09:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T03:09:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T03:09:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T03:09:46.680 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:09:46.680 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:09:46.680 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:09:46.680 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_3 2023-12-02T03:09:46.773 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T03:09:46.774 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T03:09:46.774 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T03:09:46.774 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T03:09:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T03:09:46.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T03:09:46.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T03:09:46.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:09:46.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2023-12-02T03:09:46.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:09:46.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:09:46.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:46.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_3 2023-12-02T03:09:46.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T03:09:46.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T03:09:46.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2023-12-02T03:09:46.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:09:46.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:09:46.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:46.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_3 2023-12-02T03:09:46.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-image_3.status 2023-12-02T03:09:46.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-image_3.status 2023-12-02T03:09:46.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:09:46.778 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_3.status 2023-12-02T03:09:46.778 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-image_3.status 2023-12-02T03:09:46.778 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-image_3.status 2023-12-02T03:09:46.778 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:09:46.779 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_3.status 2023-12-02T03:09:46.779 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2023-12-02T03:09:46.779 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:09:46.779 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:46.779 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_3 2023-12-02T03:09:46.779 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2023-12-02T03:09:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 123 2023-12-02T03:09:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.kebs98RNzE/cluster2-mirror-image_3.status 2023-12-02T03:09:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:09:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_3 2023-12-02T03:09:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_3.status 2023-12-02T03:09:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2023-12-02T03:09:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:09:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T03:09:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=123 2023-12-02T03:09:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:09:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:09:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T03:09:47.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_3.status 2023-12-02T03:09:47.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:09:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_3 2023-12-02T03:09:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_3.status 2023-12-02T03:09:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-02T03:09:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:09:47.637 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:09:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=70 2023-12-02T03:09:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 123 = 70 2023-12-02T03:09:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:09:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T03:09:47.940 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~97s 2023-12-02T03:09:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_3.status 2023-12-02T03:09:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:09:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:48.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_3 2023-12-02T03:09:48.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_3.status 2023-12-02T03:09:48.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-02T03:09:48.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:09:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:09:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:09:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:09:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:09:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:09:48.099 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:09:48.099 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 123 = '' 2023-12-02T03:09:48.099 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:09:48.099 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T03:09:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_3.status 2023-12-02T03:09:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:09:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_3 2023-12-02T03:09:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_3.status 2023-12-02T03:09:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-02T03:09:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:09:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:09:48.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=123 2023-12-02T03:09:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 123 = 123 2023-12-02T03:09:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:09:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2023-12-02T03:09:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:09:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_3 2023-12-02T03:09:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:09:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:09:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T03:09:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:09:49.041 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-02T03:09:49.042 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T03:09:49.042 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-02T03:09:49.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-02T03:09:49.042 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-02T03:09:49.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-02T03:09:49.045 INFO:tasks.rbd_mirror_thrash:waiting for 74 secs before thrashing 2023-12-02T03:09:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2023-12-02T03:09:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_3': 2023-12-02T03:09:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:09:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:09:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:09:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 11b8c106d4d5 2023-12-02T03:09:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.11b8c106d4d5 2023-12-02T03:09:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:09:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:09:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:09:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:09:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:00:05 2023 2023-12-02T03:09:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:00:05 2023 2023-12-02T03:09:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:00:05 2023 2023-12-02T03:09:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T03:09:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:09:50.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:09:50.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: a35ab54d-17a7-446d-984d-02da30173d31 2023-12-02T03:09:50.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:09:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:09:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror image_3 snap 2023-12-02T03:09:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_3 2023-12-02T03:09:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:09:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T03:09:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_3.export 2023-12-02T03:09:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_3.export 2023-12-02T03:09:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_3.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_3.export 2023-12-02T03:09:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.kebs98RNzE/cluster2-mirror-image_3.export 2023-12-02T03:09:50.154 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-02T03:09:50.154 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T03:09:50.154 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-02T03:09:50.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-02T03:09:50.155 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-02T03:09:50.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-02T03:09:50.158 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2023-12-02T03:09:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:09:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.kebs98RNzE/cluster1-mirror-image_3.export 2023-12-02T03:09:50.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:09:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-image_3.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_3.export 2023-12-02T03:09:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_3.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_3.export 2023-12-02T03:09:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:09:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:09:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_4 2023-12-02T03:09:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror image_4 snap 2023-12-02T03:09:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:09:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_4 2023-12-02T03:09:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:09:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T03:09:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2023-12-02T03:09:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:09:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_4 2023-12-02T03:09:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2023-12-02T03:09:50.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:09:50.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:50.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_4 2023-12-02T03:09:50.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T03:09:50.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:09:50.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:09:50.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:09:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2023-12-02T03:09:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:09:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:09:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_4 2023-12-02T03:09:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T03:09:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T03:09:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T03:09:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2023-12-02T03:09:51.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T03:09:51.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T03:09:51.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:09:51.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T03:09:51.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T03:09:51.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T03:09:51.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:09:51.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T03:09:51.651 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:09:51.651 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:09:51.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:09:51.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-12-02T03:09:51.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:09:51.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:09:51.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:09:51.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:09:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:09:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:09:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:09:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:09:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:09:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:09:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:09:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:09:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:09:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:09:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:09:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:09:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:09:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:09:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:09:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:09:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:09:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:09:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:09:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:09:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:09:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:09:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:09:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:09:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:09:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:09:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:09:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-12-02T03:09:51.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2023-12-02T03:09:51.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:09:51.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:09:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:09:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2023-12-02T03:09:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:09:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:09:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:09:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:09:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:09:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:09:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:09:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:09:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:09:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:09:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:09:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:09:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:09:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:09:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:09:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:09:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:09:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:09:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:09:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:09:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:09:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:09:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:09:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:09:51.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:09:51.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:09:51.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:09:51.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:09:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:09:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:09:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:09:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2023-12-02T03:09:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:no valid command found; 2 closest matches: 2023-12-02T03:09:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_40 2023-12-02T03:09:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_47 2023-12-02T03:09:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:09:51.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:09:51.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2023-12-02T03:09:51.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T03:09:51.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:09:51.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:09:51.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T03:09:51.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T03:09:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:09:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:09:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:09:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T03:09:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T03:09:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:09:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T03:09:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:09:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T03:09:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T03:09:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:09:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:09:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T03:09:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T03:09:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:09:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:09:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:09:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T03:09:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:09:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:09:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T03:09:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T03:09:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T03:09:51.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:09:51.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:09:51.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:09:51.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2023-12-02T03:09:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:no valid command found; 3 closest matches: 2023-12-02T03:09:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_41 2023-12-02T03:09:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_45 2023-12-02T03:09:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_49 2023-12-02T03:09:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:09:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:09:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_4 2023-12-02T03:09:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T03:09:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:09:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:09:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T03:09:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T03:09:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:09:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:09:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:09:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T03:09:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T03:09:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:09:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T03:09:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:09:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T03:09:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T03:09:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:09:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:09:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T03:09:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T03:09:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:09:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:09:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:09:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T03:09:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:09:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:09:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T03:09:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T03:09:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T03:09:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:09:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:09:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:09:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_4 2023-12-02T03:09:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:no valid command found; 4 closest matches: 2023-12-02T03:09:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_42 2023-12-02T03:09:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_43 2023-12-02T03:09:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_44 2023-12-02T03:09:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_48 2023-12-02T03:09:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:09:52.083 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:09:52.083 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 1 2023-12-02T03:09:53.084 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:09:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:09:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:09:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-12-02T03:09:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:09:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:09:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:09:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:09:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:09:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:09:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:09:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:09:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:09:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:09:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:09:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:09:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:09:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:09:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:09:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:09:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:09:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:09:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:09:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:09:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:09:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:09:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:09:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:09:53.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:09:53.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:09:53.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:09:53.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:09:53.108 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:09:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:09:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:09:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-12-02T03:09:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2023-12-02T03:09:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:09:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:09:53.195 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:09:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2023-12-02T03:09:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:09:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:09:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:09:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:09:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:09:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:09:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:09:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:09:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:09:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:09:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:09:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:09:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:09:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:09:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:09:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:09:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:09:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:09:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:09:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:09:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:09:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:09:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:09:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:09:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:09:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:09:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:09:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:09:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:09:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:09:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:09:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2023-12-02T03:09:53.302 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:no valid command found; 2 closest matches: 2023-12-02T03:09:53.302 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_40 2023-12-02T03:09:53.302 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_47 2023-12-02T03:09:53.302 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:09:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:09:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2023-12-02T03:09:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T03:09:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:09:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:09:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T03:09:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T03:09:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:09:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:09:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:09:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T03:09:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T03:09:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:09:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T03:09:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:09:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T03:09:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T03:09:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:09:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:09:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T03:09:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T03:09:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:09:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:09:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:09:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T03:09:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:09:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:09:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T03:09:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T03:09:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T03:09:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:09:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:09:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:09:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2023-12-02T03:09:53.344 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~103s 2023-12-02T03:09:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:no valid command found; 3 closest matches: 2023-12-02T03:09:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_41 2023-12-02T03:09:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_45 2023-12-02T03:09:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_49 2023-12-02T03:09:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:09:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:09:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_4 2023-12-02T03:09:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T03:09:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:09:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:09:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T03:09:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T03:09:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:09:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:09:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:09:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T03:09:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T03:09:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:09:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T03:09:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:09:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T03:09:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T03:09:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:09:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:09:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T03:09:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T03:09:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:09:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:09:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:09:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T03:09:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:09:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:09:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T03:09:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T03:09:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T03:09:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:09:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:09:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:09:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_4 2023-12-02T03:09:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:no valid command found; 4 closest matches: 2023-12-02T03:09:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_42 2023-12-02T03:09:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_43 2023-12-02T03:09:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_44 2023-12-02T03:09:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_48 2023-12-02T03:09:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:09:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:09:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 2 2023-12-02T03:09:55.522 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:09:55.523 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:09:55.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:09:55.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-12-02T03:09:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:09:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:09:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:09:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:09:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:09:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:09:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:09:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:09:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:09:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:09:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:09:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:09:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:09:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:09:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:09:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:09:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:09:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:09:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:09:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:09:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:09:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:09:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:09:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:09:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:09:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:09:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:09:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:09:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:09:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:09:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:09:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-12-02T03:09:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2023-12-02T03:09:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:09:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:09:55.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:09:55.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2023-12-02T03:09:55.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:09:55.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:09:55.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:09:55.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:09:55.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:09:55.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:09:55.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:09:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:09:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:09:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:09:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:09:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:09:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:09:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:09:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:09:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:09:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:09:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:09:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:09:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:09:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:09:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:09:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:09:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:09:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:09:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:09:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:09:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:09:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:09:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:09:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:09:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2023-12-02T03:09:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:no valid command found; 2 closest matches: 2023-12-02T03:09:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_40 2023-12-02T03:09:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_47 2023-12-02T03:09:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:09:55.741 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:09:55.741 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2023-12-02T03:09:55.741 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T03:09:55.741 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:09:55.741 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:09:55.741 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T03:09:55.741 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T03:09:55.741 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:09:55.741 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:09:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:09:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T03:09:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T03:09:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:09:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T03:09:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:09:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T03:09:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T03:09:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:09:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:09:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T03:09:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T03:09:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:09:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:09:55.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:55.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:09:55.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T03:09:55.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:09:55.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:09:55.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T03:09:55.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T03:09:55.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T03:09:55.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:09:55.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:09:55.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:09:55.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2023-12-02T03:09:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:no valid command found; 3 closest matches: 2023-12-02T03:09:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_41 2023-12-02T03:09:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_45 2023-12-02T03:09:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_49 2023-12-02T03:09:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:09:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:09:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_4 2023-12-02T03:09:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T03:09:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:09:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:09:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T03:09:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T03:09:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:09:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:09:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:09:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T03:09:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T03:09:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:09:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T03:09:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:09:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T03:09:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T03:09:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:09:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:09:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T03:09:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T03:09:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:09:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:09:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:09:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T03:09:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:09:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:09:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T03:09:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T03:09:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T03:09:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:09:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:09:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:09:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_4 2023-12-02T03:09:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:no valid command found; 4 closest matches: 2023-12-02T03:09:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_42 2023-12-02T03:09:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_43 2023-12-02T03:09:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_44 2023-12-02T03:09:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_48 2023-12-02T03:09:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:09:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:09:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 4 2023-12-02T03:09:58.748 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~108s 2023-12-02T03:09:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:09:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:09:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:09:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-12-02T03:09:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:09:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:09:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:09:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:09:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:09:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:09:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:09:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:09:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:09:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:09:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:09:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:09:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:09:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:09:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:09:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:09:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:09:59.979 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:09:59.979 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-12-02T03:10:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2023-12-02T03:10:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:10:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:10:00.069 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:10:00.069 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2023-12-02T03:10:00.069 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:10:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:10:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:10:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:10:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:10:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:10:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:10:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:10:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:10:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:10:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:10:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:10:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:10:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:10:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:10:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:10:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:10:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:10:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:10:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:10:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:10:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:10:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:10:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:10:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:10:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:10:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:10:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:10:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:10:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:10:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2023-12-02T03:10:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:no valid command found; 2 closest matches: 2023-12-02T03:10:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_40 2023-12-02T03:10:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_47 2023-12-02T03:10:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:10:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:10:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2023-12-02T03:10:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T03:10:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:10:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T03:10:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T03:10:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:10:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:10:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:10:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T03:10:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T03:10:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:10:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T03:10:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:10:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T03:10:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T03:10:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:10:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:10:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T03:10:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T03:10:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:10:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:10:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:10:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T03:10:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:10:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:10:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T03:10:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T03:10:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T03:10:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:10:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:10:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:10:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2023-12-02T03:10:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:no valid command found; 3 closest matches: 2023-12-02T03:10:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_41 2023-12-02T03:10:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_45 2023-12-02T03:10:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_49 2023-12-02T03:10:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:10:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:10:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_4 2023-12-02T03:10:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T03:10:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:10:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T03:10:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T03:10:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:10:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:10:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:10:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T03:10:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T03:10:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:10:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T03:10:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:10:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T03:10:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T03:10:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:10:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:10:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T03:10:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T03:10:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:10:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:10:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:10:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T03:10:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:10:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:10:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T03:10:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T03:10:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T03:10:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:10:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:10:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:10:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_4 2023-12-02T03:10:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:no valid command found; 4 closest matches: 2023-12-02T03:10:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_42 2023-12-02T03:10:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_43 2023-12-02T03:10:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_44 2023-12-02T03:10:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_48 2023-12-02T03:10:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:10:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:10:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 8 2023-12-02T03:10:04.152 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~113s 2023-12-02T03:10:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:10:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:10:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:10:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-12-02T03:10:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:10:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:10:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:10:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:10:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:10:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:10:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:10:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:10:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:10:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:10:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:10:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:10:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:10:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:10:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:10:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:10:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:10:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:10:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:10:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:10:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:10:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:10:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:10:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:10:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:10:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:10:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:10:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:10:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:10:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:10:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-12-02T03:10:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2023-12-02T03:10:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:10:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:10:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:10:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2023-12-02T03:10:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:10:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:10:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:10:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:10:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:10:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:10:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:10:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:10:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:10:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:10:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:10:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:10:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:10:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:10:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:10:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:10:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:10:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:10:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:10:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:10:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:10:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:10:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:10:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:10:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:10:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:10:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:10:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:10:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:10:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:10:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2023-12-02T03:10:08.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:no valid command found; 2 closest matches: 2023-12-02T03:10:08.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_40 2023-12-02T03:10:08.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_47 2023-12-02T03:10:08.610 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:10:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:10:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2023-12-02T03:10:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T03:10:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:10:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T03:10:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T03:10:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:10:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:10:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:10:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T03:10:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T03:10:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:10:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T03:10:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:10:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T03:10:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T03:10:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:10:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:10:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T03:10:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T03:10:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:10:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:10:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:10:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T03:10:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:10:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:10:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T03:10:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T03:10:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T03:10:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:10:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:10:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:10:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2023-12-02T03:10:08.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:no valid command found; 3 closest matches: 2023-12-02T03:10:08.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_41 2023-12-02T03:10:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_45 2023-12-02T03:10:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_49 2023-12-02T03:10:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:10:08.720 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:10:08.720 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_4 2023-12-02T03:10:08.720 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T03:10:08.720 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:08.720 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:10:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T03:10:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T03:10:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:10:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:10:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:10:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T03:10:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T03:10:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:10:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T03:10:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:10:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T03:10:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T03:10:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:10:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:10:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T03:10:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T03:10:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:10:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:10:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:10:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T03:10:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:10:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:10:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T03:10:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T03:10:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T03:10:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:10:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:10:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:10:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_4 2023-12-02T03:10:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:no valid command found; 4 closest matches: 2023-12-02T03:10:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_42 2023-12-02T03:10:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_43 2023-12-02T03:10:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_44 2023-12-02T03:10:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_48 2023-12-02T03:10:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:10:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:10:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 8 2023-12-02T03:10:09.556 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~119s 2023-12-02T03:10:14.960 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~124s 2023-12-02T03:10:16.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:10:16.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:10:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:10:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-12-02T03:10:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:10:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:10:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:10:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:10:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:10:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:10:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:10:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:10:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:10:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:10:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:10:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:10:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:10:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:10:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:10:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:10:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:10:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:10:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:10:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:10:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:10:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:10:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:10:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:10:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:10:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:10:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:10:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:10:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:10:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:10:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-12-02T03:10:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2023-12-02T03:10:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:10:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:10:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:10:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2023-12-02T03:10:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:10:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:10:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:10:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:10:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:10:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:10:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:10:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:10:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:10:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:10:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:10:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:10:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:10:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:10:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:10:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:10:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:10:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:10:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:10:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:10:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:10:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:10:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:10:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:10:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:10:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:10:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:10:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:10:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:10:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:10:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2023-12-02T03:10:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:no valid command found; 2 closest matches: 2023-12-02T03:10:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_40 2023-12-02T03:10:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_47 2023-12-02T03:10:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:10:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:10:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2023-12-02T03:10:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T03:10:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:10:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T03:10:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T03:10:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:10:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:10:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:10:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T03:10:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T03:10:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:10:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T03:10:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:10:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T03:10:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T03:10:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:10:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:10:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T03:10:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T03:10:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:10:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:10:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:10:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T03:10:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:10:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:10:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T03:10:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T03:10:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T03:10:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:10:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:10:17.068 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:10:17.068 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2023-12-02T03:10:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:no valid command found; 3 closest matches: 2023-12-02T03:10:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_41 2023-12-02T03:10:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_45 2023-12-02T03:10:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_49 2023-12-02T03:10:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:10:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:10:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_4 2023-12-02T03:10:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T03:10:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:10:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T03:10:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T03:10:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:10:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:10:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:10:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T03:10:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T03:10:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:10:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T03:10:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:10:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T03:10:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T03:10:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:10:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:10:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T03:10:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T03:10:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:10:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:10:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:10:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T03:10:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:10:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:10:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T03:10:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T03:10:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T03:10:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:10:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:10:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:10:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_4 2023-12-02T03:10:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:no valid command found; 4 closest matches: 2023-12-02T03:10:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_42 2023-12-02T03:10:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_43 2023-12-02T03:10:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_44 2023-12-02T03:10:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_48 2023-12-02T03:10:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:10:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:10:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 8 2023-12-02T03:10:20.363 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~130s 2023-12-02T03:10:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:10:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:10:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:10:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-12-02T03:10:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:10:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:10:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:10:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:10:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:10:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:10:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:10:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:10:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:10:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:10:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:10:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:10:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:10:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:10:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:10:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:10:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:10:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:10:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:10:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:10:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:10:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:10:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:10:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:10:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:10:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:10:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:10:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:10:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:10:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:10:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-12-02T03:10:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T03:10:25.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T03:10:25.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T03:10:25.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T03:10:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T03:10:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T03:10:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T03:10:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:10:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2023-12-02T03:10:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:10:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_4 2023-12-02T03:10:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T03:10:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T03:10:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2023-12-02T03:10:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:10:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_4 2023-12-02T03:10:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-image_4.status 2023-12-02T03:10:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-image_4.status 2023-12-02T03:10:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:10:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_4.status 2023-12-02T03:10:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-image_4.status 2023-12-02T03:10:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-image_4.status 2023-12-02T03:10:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:10:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_4.status 2023-12-02T03:10:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2023-12-02T03:10:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_4 2023-12-02T03:10:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2023-12-02T03:10:25.767 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~135s 2023-12-02T03:10:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 125 2023-12-02T03:10:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.kebs98RNzE/cluster2-mirror-image_4.status 2023-12-02T03:10:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_4 2023-12-02T03:10:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_4.status 2023-12-02T03:10:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2023-12-02T03:10:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:10:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T03:10:26.322 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=125 2023-12-02T03:10:26.322 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:10:26.322 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:10:26.322 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T03:10:26.523 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_4.status 2023-12-02T03:10:26.523 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:26.523 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_4 2023-12-02T03:10:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_4.status 2023-12-02T03:10:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-02T03:10:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:10:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:10:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:10:26.572 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:10:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:10:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:10:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:10:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 125 = '' 2023-12-02T03:10:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:10:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T03:10:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_4.status 2023-12-02T03:10:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_4 2023-12-02T03:10:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_4.status 2023-12-02T03:10:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-02T03:10:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:10:27.024 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:10:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:10:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:10:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:10:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:10:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:10:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 125 = '' 2023-12-02T03:10:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:10:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T03:10:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_4.status 2023-12-02T03:10:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_4 2023-12-02T03:10:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_4.status 2023-12-02T03:10:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-02T03:10:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:10:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:10:27.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=125 2023-12-02T03:10:27.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 125 = 125 2023-12-02T03:10:27.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:10:27.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2023-12-02T03:10:27.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:27.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:27.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_4 2023-12-02T03:10:27.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:10:27.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:10:27.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T03:10:27.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:10:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2023-12-02T03:10:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_4': 2023-12-02T03:10:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:10:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:10:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:10:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 121175fc59c5 2023-12-02T03:10:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.121175fc59c5 2023-12-02T03:10:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:10:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:10:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:10:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:10:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:00:15 2023 2023-12-02T03:10:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:00:15 2023 2023-12-02T03:10:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:00:15 2023 2023-12-02T03:10:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T03:10:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:10:28.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:10:28.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 730af9f3-f6b6-4cf2-9474-5684c31f584d 2023-12-02T03:10:28.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:10:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:10:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror image_4 snap 2023-12-02T03:10:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_4 2023-12-02T03:10:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:10:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T03:10:28.930 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_4.export 2023-12-02T03:10:28.930 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_4.export 2023-12-02T03:10:28.930 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_4.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_4.export 2023-12-02T03:10:28.930 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.kebs98RNzE/cluster2-mirror-image_4.export 2023-12-02T03:10:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:10:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.kebs98RNzE/cluster1-mirror-image_4.export 2023-12-02T03:10:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:10:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-image_4.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_4.export 2023-12-02T03:10:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_4.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_4.export 2023-12-02T03:10:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:10:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:10:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_5 2023-12-02T03:10:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror image_5 snap 2023-12-02T03:10:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_5 2023-12-02T03:10:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:10:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T03:10:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2023-12-02T03:10:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_5 2023-12-02T03:10:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2023-12-02T03:10:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_5 2023-12-02T03:10:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T03:10:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:10:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:10:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:10:31.170 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~140s 2023-12-02T03:10:31.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2023-12-02T03:10:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_5 2023-12-02T03:10:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T03:10:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T03:10:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T03:10:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2023-12-02T03:10:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T03:10:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T03:10:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T03:10:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T03:10:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T03:10:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:10:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T03:10:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:10:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:10:31.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:10:31.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2023-12-02T03:10:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:10:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:10:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:10:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:10:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:10:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:10:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:10:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:10:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:10:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:10:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:10:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:10:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:10:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:10:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:10:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:10:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:10:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:10:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:10:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:10:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:10:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:10:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:10:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:10:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:10:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:10:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:10:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:10:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:10:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:10:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2023-12-02T03:10:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2023-12-02T03:10:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:10:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:10:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:10:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2023-12-02T03:10:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:10:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:10:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:10:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:10:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:10:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:10:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:10:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:10:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:10:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:10:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:10:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:10:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:10:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:10:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:10:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:10:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:10:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:10:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:10:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:10:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:10:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:10:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:10:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:10:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:10:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:10:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:10:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:10:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:10:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:10:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2023-12-02T03:10:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2023-12-02T03:10:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:10:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:10:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:10:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2023-12-02T03:10:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T03:10:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:10:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T03:10:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T03:10:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:10:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:10:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:10:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T03:10:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T03:10:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:10:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:31.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T03:10:31.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:10:31.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T03:10:31.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T03:10:31.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:10:31.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:10:31.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T03:10:31.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T03:10:31.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:10:31.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:10:31.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:31.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:10:31.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T03:10:31.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:10:31.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:10:31.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T03:10:31.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T03:10:31.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T03:10:31.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:10:31.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:10:31.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:10:31.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2023-12-02T03:10:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:no valid command found; 1 closest matches: 2023-12-02T03:10:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status mirror/image_50 2023-12-02T03:10:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:10:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:10:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_5 2023-12-02T03:10:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T03:10:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:10:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T03:10:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T03:10:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:10:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:10:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:10:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T03:10:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T03:10:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:10:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T03:10:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:10:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T03:10:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T03:10:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:10:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:10:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T03:10:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T03:10:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:10:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:10:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:10:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T03:10:31.609 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:10:31.609 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:10:31.609 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T03:10:31.609 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T03:10:31.609 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T03:10:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:10:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:10:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:10:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_5 2023-12-02T03:10:31.718 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T03:10:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T03:10:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T03:10:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T03:10:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T03:10:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T03:10:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T03:10:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:10:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2023-12-02T03:10:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:10:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_5 2023-12-02T03:10:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T03:10:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T03:10:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2023-12-02T03:10:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:10:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_5 2023-12-02T03:10:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-image_5.status 2023-12-02T03:10:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-image_5.status 2023-12-02T03:10:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:10:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_5.status 2023-12-02T03:10:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-image_5.status 2023-12-02T03:10:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-image_5.status 2023-12-02T03:10:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:10:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_5.status 2023-12-02T03:10:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2023-12-02T03:10:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_5 2023-12-02T03:10:31.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2023-12-02T03:10:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 127 2023-12-02T03:10:32.281 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.kebs98RNzE/cluster2-mirror-image_5.status 2023-12-02T03:10:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_5 2023-12-02T03:10:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_5.status 2023-12-02T03:10:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2023-12-02T03:10:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:10:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T03:10:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=127 2023-12-02T03:10:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:10:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:10:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T03:10:32.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_5.status 2023-12-02T03:10:32.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:32.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:32.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_5 2023-12-02T03:10:32.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_5.status 2023-12-02T03:10:32.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-02T03:10:32.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:10:32.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:10:32.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:10:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:10:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:10:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:10:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:10:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 127 = '' 2023-12-02T03:10:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:10:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T03:10:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_5.status 2023-12-02T03:10:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_5 2023-12-02T03:10:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_5.status 2023-12-02T03:10:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-02T03:10:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:10:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:10:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:10:33.038 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:10:33.038 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:10:33.038 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:10:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:10:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 127 = '' 2023-12-02T03:10:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:10:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T03:10:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_5.status 2023-12-02T03:10:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_5 2023-12-02T03:10:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_5.status 2023-12-02T03:10:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-02T03:10:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:10:33.885 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:10:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=127 2023-12-02T03:10:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 127 = 127 2023-12-02T03:10:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:10:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2023-12-02T03:10:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_5 2023-12-02T03:10:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:10:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:10:33.888 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T03:10:33.888 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:10:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2023-12-02T03:10:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_5': 2023-12-02T03:10:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:10:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:10:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:10:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12a1cb5e484e 2023-12-02T03:10:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12a1cb5e484e 2023-12-02T03:10:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:10:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:10:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:10:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:10:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:00:27 2023 2023-12-02T03:10:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:00:27 2023 2023-12-02T03:10:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:00:27 2023 2023-12-02T03:10:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T03:10:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:10:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:10:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 19ecaa0e-982b-435c-a3a7-d881b478cee6 2023-12-02T03:10:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:10:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:10:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror image_5 snap 2023-12-02T03:10:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_5 2023-12-02T03:10:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:10:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T03:10:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_5.export 2023-12-02T03:10:34.940 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_5.export 2023-12-02T03:10:34.940 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_5.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_5.export 2023-12-02T03:10:34.940 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.kebs98RNzE/cluster2-mirror-image_5.export 2023-12-02T03:10:35.129 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:10:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.kebs98RNzE/cluster1-mirror-image_5.export 2023-12-02T03:10:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:10:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-image_5.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_5.export 2023-12-02T03:10:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_5.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_5.export 2023-12-02T03:10:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:10:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:10:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_6 2023-12-02T03:10:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror image_6 snap 2023-12-02T03:10:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_6 2023-12-02T03:10:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:10:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T03:10:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2023-12-02T03:10:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_6 2023-12-02T03:10:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2023-12-02T03:10:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_6 2023-12-02T03:10:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T03:10:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:10:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:10:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:10:36.575 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~146s 2023-12-02T03:10:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2023-12-02T03:10:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_6 2023-12-02T03:10:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T03:10:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T03:10:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T03:10:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2023-12-02T03:10:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T03:10:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T03:10:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T03:10:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T03:10:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T03:10:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:10:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T03:10:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:10:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:10:37.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:10:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2023-12-02T03:10:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:10:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:10:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:10:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:10:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:10:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:10:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:10:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:10:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:10:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:10:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:10:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:10:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:10:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:10:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:10:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:10:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:10:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:10:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:10:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:10:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:10:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:10:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:10:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:10:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:10:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:10:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:10:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:10:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:10:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:10:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2023-12-02T03:10:37.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2023-12-02T03:10:37.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:10:37.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:10:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:10:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2023-12-02T03:10:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:10:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:10:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:10:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:10:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:10:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:10:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:10:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:10:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:10:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:10:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:10:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:10:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:10:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:10:37.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:10:37.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:10:37.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:10:37.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:10:37.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:10:37.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:10:37.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:37.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:10:37.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:10:37.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:10:37.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:10:37.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:10:37.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:10:37.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:10:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:10:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:10:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:10:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2023-12-02T03:10:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2023-12-02T03:10:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:10:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:10:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:10:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_6 2023-12-02T03:10:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T03:10:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:10:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T03:10:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T03:10:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:10:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:10:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:10:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T03:10:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T03:10:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:10:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T03:10:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:10:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T03:10:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T03:10:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:10:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:10:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T03:10:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T03:10:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:10:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:10:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:10:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T03:10:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:10:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:10:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T03:10:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T03:10:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T03:10:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:10:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:10:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:10:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_6 2023-12-02T03:10:37.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T03:10:37.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T03:10:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T03:10:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T03:10:37.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T03:10:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T03:10:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T03:10:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:10:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2023-12-02T03:10:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:10:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_6 2023-12-02T03:10:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T03:10:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T03:10:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2023-12-02T03:10:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:10:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_6 2023-12-02T03:10:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-image_6.status 2023-12-02T03:10:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-image_6.status 2023-12-02T03:10:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:10:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_6.status 2023-12-02T03:10:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-image_6.status 2023-12-02T03:10:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-image_6.status 2023-12-02T03:10:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:10:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_6.status 2023-12-02T03:10:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2023-12-02T03:10:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_6 2023-12-02T03:10:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2023-12-02T03:10:38.335 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 129 2023-12-02T03:10:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.kebs98RNzE/cluster2-mirror-image_6.status 2023-12-02T03:10:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_6 2023-12-02T03:10:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_6.status 2023-12-02T03:10:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2023-12-02T03:10:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:10:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T03:10:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=129 2023-12-02T03:10:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:10:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:10:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T03:10:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_6.status 2023-12-02T03:10:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_6 2023-12-02T03:10:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_6.status 2023-12-02T03:10:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-02T03:10:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:10:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:10:38.648 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:10:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:10:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:10:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:10:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:10:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 129 = '' 2023-12-02T03:10:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:10:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T03:10:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_6.status 2023-12-02T03:10:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_6 2023-12-02T03:10:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_6.status 2023-12-02T03:10:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-02T03:10:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:10:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:10:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:10:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:10:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:10:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:10:39.106 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:10:39.106 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 129 = '' 2023-12-02T03:10:39.106 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:10:39.106 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T03:10:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_6.status 2023-12-02T03:10:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_6 2023-12-02T03:10:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_6.status 2023-12-02T03:10:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-02T03:10:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:10:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:10:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=129 2023-12-02T03:10:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 129 = 129 2023-12-02T03:10:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:10:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2023-12-02T03:10:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_6 2023-12-02T03:10:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:10:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:10:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T03:10:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:10:40.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2023-12-02T03:10:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_6': 2023-12-02T03:10:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:10:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:10:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:10:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 11b85bdab31 2023-12-02T03:10:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.11b85bdab31 2023-12-02T03:10:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:10:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:10:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:10:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:10:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:00:36 2023 2023-12-02T03:10:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:00:36 2023 2023-12-02T03:10:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:00:36 2023 2023-12-02T03:10:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T03:10:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:10:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:10:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: f1a9b266-fc3d-4246-8a57-fe1695af4b4d 2023-12-02T03:10:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:10:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:10:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror image_6 snap 2023-12-02T03:10:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:41.006 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_6 2023-12-02T03:10:41.006 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:10:41.006 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T03:10:41.006 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_6.export 2023-12-02T03:10:41.006 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_6.export 2023-12-02T03:10:41.006 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_6.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_6.export 2023-12-02T03:10:41.006 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.kebs98RNzE/cluster2-mirror-image_6.export 2023-12-02T03:10:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:10:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.kebs98RNzE/cluster1-mirror-image_6.export 2023-12-02T03:10:41.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:10:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-image_6.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_6.export 2023-12-02T03:10:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_6.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_6.export 2023-12-02T03:10:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:10:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:10:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_7 2023-12-02T03:10:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror image_7 snap 2023-12-02T03:10:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_7 2023-12-02T03:10:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:10:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T03:10:41.979 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~151s 2023-12-02T03:10:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2023-12-02T03:10:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_7 2023-12-02T03:10:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2023-12-02T03:10:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_7 2023-12-02T03:10:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T03:10:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:10:42.345 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:10:42.345 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:10:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2023-12-02T03:10:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_7 2023-12-02T03:10:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T03:10:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T03:10:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T03:10:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2023-12-02T03:10:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T03:10:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T03:10:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T03:10:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T03:10:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T03:10:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:10:43.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T03:10:43.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:10:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:10:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:10:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2023-12-02T03:10:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:10:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:10:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:10:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:10:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:10:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:10:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:10:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:10:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:10:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:10:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:10:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:10:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:10:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:10:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:10:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:10:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:10:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:10:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:10:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:10:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:10:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:10:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:10:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:10:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:10:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:10:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:10:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:10:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:10:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:10:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2023-12-02T03:10:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-12-02T03:10:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:10:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:10:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:10:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2023-12-02T03:10:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:10:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:10:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:10:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:10:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:10:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:10:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:10:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:10:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:10:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:10:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:10:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:10:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:10:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:10:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:10:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:10:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:10:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:10:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:10:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:10:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:10:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:10:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:10:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:10:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:10:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:10:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:10:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:10:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:10:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:10:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2023-12-02T03:10:43.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-12-02T03:10:43.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:10:43.576 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:10:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:10:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_7 2023-12-02T03:10:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T03:10:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:10:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T03:10:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T03:10:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:10:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:10:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:10:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T03:10:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T03:10:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:10:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T03:10:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:10:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T03:10:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T03:10:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:10:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:10:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T03:10:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T03:10:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:10:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:10:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:10:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T03:10:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:10:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:10:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T03:10:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T03:10:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T03:10:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:10:43.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:10:43.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:10:43.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_7 2023-12-02T03:10:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-12-02T03:10:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:10:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:10:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:10:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_7 2023-12-02T03:10:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T03:10:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:10:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T03:10:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T03:10:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:10:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:10:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:10:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T03:10:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T03:10:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:10:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T03:10:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:10:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T03:10:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T03:10:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:10:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:10:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T03:10:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T03:10:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:10:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:10:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:10:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T03:10:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:10:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:10:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T03:10:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T03:10:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T03:10:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:10:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:10:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:10:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_7 2023-12-02T03:10:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T03:10:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T03:10:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T03:10:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T03:10:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T03:10:43.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T03:10:43.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T03:10:43.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:10:43.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2023-12-02T03:10:43.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:10:43.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:43.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_7 2023-12-02T03:10:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T03:10:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T03:10:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2023-12-02T03:10:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:10:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_7 2023-12-02T03:10:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-image_7.status 2023-12-02T03:10:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-image_7.status 2023-12-02T03:10:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:10:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_7.status 2023-12-02T03:10:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-image_7.status 2023-12-02T03:10:43.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-image_7.status 2023-12-02T03:10:43.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:10:43.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_7.status 2023-12-02T03:10:43.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2023-12-02T03:10:43.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:43.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:43.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_7 2023-12-02T03:10:43.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2023-12-02T03:10:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 131 2023-12-02T03:10:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.kebs98RNzE/cluster2-mirror-image_7.status 2023-12-02T03:10:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_7 2023-12-02T03:10:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_7.status 2023-12-02T03:10:44.359 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2023-12-02T03:10:44.359 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:10:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T03:10:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=131 2023-12-02T03:10:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:10:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:10:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T03:10:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_7.status 2023-12-02T03:10:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_7 2023-12-02T03:10:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_7.status 2023-12-02T03:10:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-02T03:10:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:10:44.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:10:44.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:10:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:10:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:10:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:10:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:10:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 131 = '' 2023-12-02T03:10:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:10:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T03:10:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_7.status 2023-12-02T03:10:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_7 2023-12-02T03:10:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_7.status 2023-12-02T03:10:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-02T03:10:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:10:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:10:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:10:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:10:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:10:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:10:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:10:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 131 = '' 2023-12-02T03:10:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:10:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T03:10:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_7.status 2023-12-02T03:10:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_7 2023-12-02T03:10:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_7.status 2023-12-02T03:10:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-02T03:10:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:10:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:10:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=131 2023-12-02T03:10:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 131 = 131 2023-12-02T03:10:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:10:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2023-12-02T03:10:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_7 2023-12-02T03:10:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:10:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:10:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T03:10:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:10:46.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2023-12-02T03:10:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_7': 2023-12-02T03:10:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:10:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:10:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:10:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 121182e7c028 2023-12-02T03:10:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.121182e7c028 2023-12-02T03:10:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:10:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:10:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:10:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:10:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:00:47 2023 2023-12-02T03:10:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:00:47 2023 2023-12-02T03:10:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:00:47 2023 2023-12-02T03:10:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T03:10:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:10:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:10:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 14b5094b-8df2-410f-b1dd-ae91594e9848 2023-12-02T03:10:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:10:47.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:10:47.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror image_7 snap 2023-12-02T03:10:47.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:47.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_7 2023-12-02T03:10:47.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:10:47.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T03:10:47.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_7.export 2023-12-02T03:10:47.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_7.export 2023-12-02T03:10:47.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_7.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_7.export 2023-12-02T03:10:47.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.kebs98RNzE/cluster2-mirror-image_7.export 2023-12-02T03:10:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:10:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.kebs98RNzE/cluster1-mirror-image_7.export 2023-12-02T03:10:47.382 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~157s 2023-12-02T03:10:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:10:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-image_7.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_7.export 2023-12-02T03:10:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_7.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_7.export 2023-12-02T03:10:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:10:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:10:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_8 2023-12-02T03:10:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror image_8 snap 2023-12-02T03:10:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_8 2023-12-02T03:10:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:10:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T03:10:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2023-12-02T03:10:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_8 2023-12-02T03:10:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2023-12-02T03:10:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_8 2023-12-02T03:10:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T03:10:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:10:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:10:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:10:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2023-12-02T03:10:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_8 2023-12-02T03:10:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T03:10:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T03:10:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T03:10:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2023-12-02T03:10:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T03:10:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T03:10:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T03:10:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T03:10:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T03:10:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:10:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T03:10:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:10:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:10:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:10:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2023-12-02T03:10:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:10:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:10:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:10:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:10:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:10:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:10:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:10:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:10:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:10:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:10:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:10:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:10:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:10:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:10:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:10:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:10:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:10:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:10:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:10:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:10:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:10:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:10:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:10:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:10:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:10:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:10:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:10:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:10:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:10:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:10:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2023-12-02T03:10:49.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T03:10:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T03:10:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T03:10:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T03:10:49.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T03:10:49.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T03:10:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T03:10:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:10:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2023-12-02T03:10:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:10:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_8 2023-12-02T03:10:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T03:10:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T03:10:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2023-12-02T03:10:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:10:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_8 2023-12-02T03:10:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-image_8.status 2023-12-02T03:10:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-image_8.status 2023-12-02T03:10:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:10:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_8.status 2023-12-02T03:10:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-image_8.status 2023-12-02T03:10:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-image_8.status 2023-12-02T03:10:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:10:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_8.status 2023-12-02T03:10:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2023-12-02T03:10:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_8 2023-12-02T03:10:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2023-12-02T03:10:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 133 2023-12-02T03:10:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.kebs98RNzE/cluster2-mirror-image_8.status 2023-12-02T03:10:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_8 2023-12-02T03:10:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_8.status 2023-12-02T03:10:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2023-12-02T03:10:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:10:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T03:10:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=133 2023-12-02T03:10:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:10:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:10:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T03:10:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_8.status 2023-12-02T03:10:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_8 2023-12-02T03:10:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_8.status 2023-12-02T03:10:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-02T03:10:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:10:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:10:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:10:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:10:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:10:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:10:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:10:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 133 = '' 2023-12-02T03:10:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:10:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T03:10:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_8.status 2023-12-02T03:10:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_8 2023-12-02T03:10:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_8.status 2023-12-02T03:10:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-02T03:10:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:10:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:10:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:10:51.146 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:10:51.147 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:10:51.147 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:10:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:10:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 133 = '' 2023-12-02T03:10:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:10:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T03:10:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_8.status 2023-12-02T03:10:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_8 2023-12-02T03:10:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_8.status 2023-12-02T03:10:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-02T03:10:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:10:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:10:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=133 2023-12-02T03:10:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 133 = 133 2023-12-02T03:10:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:10:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2023-12-02T03:10:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:51.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_8 2023-12-02T03:10:51.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:10:51.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:10:51.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T03:10:51.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:10:52.785 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~162s 2023-12-02T03:10:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2023-12-02T03:10:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_8': 2023-12-02T03:10:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:10:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:10:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:10:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12a15bf374b1 2023-12-02T03:10:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12a15bf374b1 2023-12-02T03:10:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:10:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:10:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:10:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:10:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:00:58 2023 2023-12-02T03:10:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:00:58 2023 2023-12-02T03:10:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:00:58 2023 2023-12-02T03:10:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T03:10:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:10:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:10:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: f71e0710-9762-4b0b-8375-4f15e3c521f1 2023-12-02T03:10:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:10:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:10:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror image_8 snap 2023-12-02T03:10:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_8 2023-12-02T03:10:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:10:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T03:10:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_8.export 2023-12-02T03:10:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_8.export 2023-12-02T03:10:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_8.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_8.export 2023-12-02T03:10:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.kebs98RNzE/cluster2-mirror-image_8.export 2023-12-02T03:10:53.235 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:10:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.kebs98RNzE/cluster1-mirror-image_8.export 2023-12-02T03:10:53.493 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:10:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-image_8.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_8.export 2023-12-02T03:10:53.562 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_8.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_8.export 2023-12-02T03:10:53.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:10:53.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:10:53.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_9 2023-12-02T03:10:53.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror image_9 snap 2023-12-02T03:10:53.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:53.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:53.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_9 2023-12-02T03:10:53.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:10:53.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T03:10:54.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2023-12-02T03:10:54.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:54.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_9 2023-12-02T03:10:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2023-12-02T03:10:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_9 2023-12-02T03:10:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T03:10:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:10:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:10:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:10:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2023-12-02T03:10:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_9 2023-12-02T03:10:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T03:10:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T03:10:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T03:10:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2023-12-02T03:10:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T03:10:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T03:10:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T03:10:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T03:10:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T03:10:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:10:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T03:10:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:10:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:10:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:10:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2023-12-02T03:10:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:10:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:10:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:10:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:10:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:10:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:10:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:10:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:10:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:10:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:10:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:10:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:10:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:10:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:10:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:10:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:10:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:10:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:10:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:10:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:10:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:10:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:10:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:10:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:10:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:10:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:10:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:10:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:10:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:10:55.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:10:55.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:10:55.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:10:55.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2023-12-02T03:10:55.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T03:10:55.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T03:10:55.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T03:10:55.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T03:10:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T03:10:55.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T03:10:55.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T03:10:55.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:10:55.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2023-12-02T03:10:55.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:10:55.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:55.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:55.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_9 2023-12-02T03:10:55.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T03:10:55.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T03:10:55.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2023-12-02T03:10:55.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:10:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_9 2023-12-02T03:10:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-image_9.status 2023-12-02T03:10:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-image_9.status 2023-12-02T03:10:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:10:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_9.status 2023-12-02T03:10:55.530 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-image_9.status 2023-12-02T03:10:55.530 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-image_9.status 2023-12-02T03:10:55.530 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:10:55.530 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_9.status 2023-12-02T03:10:55.531 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2023-12-02T03:10:55.531 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:55.531 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:55.531 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_9 2023-12-02T03:10:55.531 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2023-12-02T03:10:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 135 2023-12-02T03:10:56.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.kebs98RNzE/cluster2-mirror-image_9.status 2023-12-02T03:10:56.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:56.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:56.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_9 2023-12-02T03:10:56.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_9.status 2023-12-02T03:10:56.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2023-12-02T03:10:56.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:10:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T03:10:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=135 2023-12-02T03:10:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:10:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:10:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T03:10:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_9.status 2023-12-02T03:10:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_9 2023-12-02T03:10:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_9.status 2023-12-02T03:10:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-02T03:10:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:10:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:10:56.729 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:10:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:10:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:10:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:10:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:10:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 135 = '' 2023-12-02T03:10:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:10:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T03:10:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_9.status 2023-12-02T03:10:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_9 2023-12-02T03:10:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_9.status 2023-12-02T03:10:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-02T03:10:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:10:57.186 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:10:57.186 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:10:57.187 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:10:57.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:10:57.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:10:57.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:10:57.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 135 = '' 2023-12-02T03:10:57.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:10:57.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T03:10:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_9.status 2023-12-02T03:10:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_9 2023-12-02T03:10:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_9.status 2023-12-02T03:10:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-02T03:10:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:10:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:10:58.040 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=135 2023-12-02T03:10:58.040 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 135 = 135 2023-12-02T03:10:58.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:10:58.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2023-12-02T03:10:58.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:10:58.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:58.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_9 2023-12-02T03:10:58.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:10:58.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:10:58.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T03:10:58.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:10:58.189 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~168s 2023-12-02T03:10:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2023-12-02T03:10:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_9': 2023-12-02T03:10:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:10:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:10:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:10:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 10e49b249977 2023-12-02T03:10:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.10e49b249977 2023-12-02T03:10:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:10:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:10:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:10:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:10:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:01:13 2023 2023-12-02T03:10:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:01:13 2023 2023-12-02T03:10:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:01:13 2023 2023-12-02T03:10:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T03:10:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:10:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:10:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: da41f19a-67cd-4518-a939-22e8fe4b6971 2023-12-02T03:10:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:10:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:10:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror image_9 snap 2023-12-02T03:10:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_9 2023-12-02T03:10:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:10:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T03:10:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_9.export 2023-12-02T03:10:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_9.export 2023-12-02T03:10:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_9.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_9.export 2023-12-02T03:10:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.kebs98RNzE/cluster2-mirror-image_9.export 2023-12-02T03:10:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:10:59.288 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.kebs98RNzE/cluster1-mirror-image_9.export 2023-12-02T03:10:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:10:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-image_9.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_9.export 2023-12-02T03:10:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_9.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_9.export 2023-12-02T03:10:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:10:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:10:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_10 2023-12-02T03:10:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror image_10 snap 2023-12-02T03:10:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:10:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:10:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_10 2023-12-02T03:10:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:10:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T03:11:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2023-12-02T03:11:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_10 2023-12-02T03:11:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2023-12-02T03:11:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_10 2023-12-02T03:11:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T03:11:00.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:11:00.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:11:00.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:11:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2023-12-02T03:11:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_10 2023-12-02T03:11:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T03:11:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T03:11:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T03:11:01.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2023-12-02T03:11:01.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T03:11:01.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:01.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T03:11:01.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T03:11:01.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T03:11:01.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T03:11:01.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:11:01.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T03:11:01.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:11:01.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:11:01.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:11:01.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2023-12-02T03:11:01.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:11:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:11:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:11:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:11:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:11:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:11:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:11:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:11:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:11:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:11:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:11:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:11:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:11:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:11:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:11:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:11:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:11:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:11:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:11:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:11:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:11:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:11:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:11:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:11:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:11:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:11:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:11:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:11:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:11:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:11:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2023-12-02T03:11:01.795 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-12-02T03:11:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:11:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:11:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:11:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2023-12-02T03:11:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:11:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:11:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:11:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:11:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:11:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:11:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:11:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:11:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:11:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:11:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:11:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:11:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:11:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:11:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:11:01.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:11:01.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:11:01.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:11:01.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:11:01.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:11:01.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:01.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:11:01.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:11:01.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:11:01.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:11:01.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:11:01.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:11:01.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:11:01.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:11:01.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:11:01.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:11:01.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2023-12-02T03:11:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-12-02T03:11:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:11:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:11:01.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:11:01.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_10 2023-12-02T03:11:01.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T03:11:01.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:01.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:11:01.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T03:11:01.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T03:11:01.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:11:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:11:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:11:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T03:11:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T03:11:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:11:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T03:11:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:11:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T03:11:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T03:11:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:11:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:11:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T03:11:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T03:11:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:11:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:11:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:11:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T03:11:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:11:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:11:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T03:11:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T03:11:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T03:11:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:11:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:11:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:11:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_10 2023-12-02T03:11:02.012 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-12-02T03:11:02.012 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:11:02.012 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:11:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:11:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_10 2023-12-02T03:11:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T03:11:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:11:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T03:11:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T03:11:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:11:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:11:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:11:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T03:11:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T03:11:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:11:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T03:11:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:11:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T03:11:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T03:11:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:11:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:11:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T03:11:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T03:11:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:11:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:11:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:11:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T03:11:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:11:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:11:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T03:11:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T03:11:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T03:11:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:11:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:11:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:11:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_10 2023-12-02T03:11:02.125 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T03:11:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T03:11:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T03:11:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T03:11:02.127 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T03:11:02.128 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T03:11:02.128 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T03:11:02.128 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:02.128 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2023-12-02T03:11:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:11:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_10 2023-12-02T03:11:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T03:11:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T03:11:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2023-12-02T03:11:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:11:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_10 2023-12-02T03:11:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-image_10.status 2023-12-02T03:11:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-image_10.status 2023-12-02T03:11:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:11:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_10.status 2023-12-02T03:11:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-image_10.status 2023-12-02T03:11:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-image_10.status 2023-12-02T03:11:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:11:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_10.status 2023-12-02T03:11:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2023-12-02T03:11:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_10 2023-12-02T03:11:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2023-12-02T03:11:02.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 137 2023-12-02T03:11:02.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.kebs98RNzE/cluster2-mirror-image_10.status 2023-12-02T03:11:02.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:02.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:02.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_10 2023-12-02T03:11:02.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_10.status 2023-12-02T03:11:02.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2023-12-02T03:11:02.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T03:11:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=137 2023-12-02T03:11:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:11:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T03:11:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_10.status 2023-12-02T03:11:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_10 2023-12-02T03:11:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_10.status 2023-12-02T03:11:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-02T03:11:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:03.045 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5693140605729405) > weight (0.25) 2023-12-02T03:11:03.046 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5641558045561487) > weight (0.25) 2023-12-02T03:11:03.046 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7613288360877732) > weight (0.25) 2023-12-02T03:11:03.046 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9492321331288686) > weight (0.25) 2023-12-02T03:11:03.046 INFO:tasks.rbd_mirror_thrash:waiting for 85 secs before thrashing 2023-12-02T03:11:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=77 2023-12-02T03:11:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 137 = 77 2023-12-02T03:11:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T03:11:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_10.status 2023-12-02T03:11:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_10 2023-12-02T03:11:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_10.status 2023-12-02T03:11:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-02T03:11:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:03.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~173s 2023-12-02T03:11:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:11:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:11:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:11:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:11:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:11:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 137 = '' 2023-12-02T03:11:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T03:11:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_10.status 2023-12-02T03:11:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_10 2023-12-02T03:11:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_10.status 2023-12-02T03:11:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-02T03:11:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:11:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:11:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:11:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:11:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:11:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 137 = '' 2023-12-02T03:11:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1.6 2023-12-02T03:11:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_10.status 2023-12-02T03:11:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_10 2023-12-02T03:11:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_10.status 2023-12-02T03:11:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-02T03:11:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:06.157 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=137 2023-12-02T03:11:06.157 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 137 = 137 2023-12-02T03:11:06.157 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2023-12-02T03:11:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_10 2023-12-02T03:11:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:11:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:11:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T03:11:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:11:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2023-12-02T03:11:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_10': 2023-12-02T03:11:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:11:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:11:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:11:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 11b8b21d95bc 2023-12-02T03:11:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.11b8b21d95bc 2023-12-02T03:11:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:11:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:11:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:11:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:11:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:01:25 2023 2023-12-02T03:11:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:01:25 2023 2023-12-02T03:11:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:01:25 2023 2023-12-02T03:11:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T03:11:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:11:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:11:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: ddeae318-0d59-4bf2-b55e-a3f9934ab1b6 2023-12-02T03:11:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:11:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror image_10 snap 2023-12-02T03:11:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_10 2023-12-02T03:11:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:11:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T03:11:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_10.export 2023-12-02T03:11:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_10.export 2023-12-02T03:11:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_10.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_10.export 2023-12-02T03:11:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.kebs98RNzE/cluster2-mirror-image_10.export 2023-12-02T03:11:07.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:11:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.kebs98RNzE/cluster1-mirror-image_10.export 2023-12-02T03:11:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:11:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-image_10.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_10.export 2023-12-02T03:11:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_10.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_10.export 2023-12-02T03:11:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:11:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_11 2023-12-02T03:11:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror image_11 snap 2023-12-02T03:11:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_11 2023-12-02T03:11:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:11:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T03:11:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2023-12-02T03:11:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_11 2023-12-02T03:11:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2023-12-02T03:11:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_11 2023-12-02T03:11:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T03:11:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:11:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:11:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:11:08.997 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~178s 2023-12-02T03:11:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2023-12-02T03:11:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_11 2023-12-02T03:11:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T03:11:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T03:11:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T03:11:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2023-12-02T03:11:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T03:11:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T03:11:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T03:11:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T03:11:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T03:11:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:11:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T03:11:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:11:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:11:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:11:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2023-12-02T03:11:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:11:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:11:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:11:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:11:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:11:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:11:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:11:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:11:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:11:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:11:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:11:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:11:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:11:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:11:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:11:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:11:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:11:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:11:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:11:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:11:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:11:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:11:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:11:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:11:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:11:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:11:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:11:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:11:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:11:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:11:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2023-12-02T03:11:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2023-12-02T03:11:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:11:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:11:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:11:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2023-12-02T03:11:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:11:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:11:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:11:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:11:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:11:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:11:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:11:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:11:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:11:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:11:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:11:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:11:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:11:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:11:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:11:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:11:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:11:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:11:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:11:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:11:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:11:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:11:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:11:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:11:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:11:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:11:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:11:09.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:11:09.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:11:09.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:11:09.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2023-12-02T03:11:09.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2023-12-02T03:11:09.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:11:09.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:11:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:11:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_11 2023-12-02T03:11:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T03:11:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:11:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T03:11:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T03:11:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:11:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:11:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:11:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T03:11:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T03:11:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:11:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T03:11:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:11:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T03:11:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T03:11:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:11:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:11:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T03:11:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T03:11:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:11:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:11:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:11:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T03:11:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:11:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:11:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T03:11:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T03:11:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T03:11:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:11:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:11:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:11:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_11 2023-12-02T03:11:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T03:11:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T03:11:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T03:11:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T03:11:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T03:11:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T03:11:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T03:11:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2023-12-02T03:11:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:11:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_11 2023-12-02T03:11:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T03:11:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T03:11:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2023-12-02T03:11:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:11:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_11 2023-12-02T03:11:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-image_11.status 2023-12-02T03:11:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-image_11.status 2023-12-02T03:11:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:11:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_11.status 2023-12-02T03:11:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-image_11.status 2023-12-02T03:11:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-image_11.status 2023-12-02T03:11:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:11:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_11.status 2023-12-02T03:11:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2023-12-02T03:11:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_11 2023-12-02T03:11:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2023-12-02T03:11:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 139 2023-12-02T03:11:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.kebs98RNzE/cluster2-mirror-image_11.status 2023-12-02T03:11:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_11 2023-12-02T03:11:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_11.status 2023-12-02T03:11:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2023-12-02T03:11:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:10.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T03:11:10.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=139 2023-12-02T03:11:10.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:11:10.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:10.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T03:11:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_11.status 2023-12-02T03:11:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_11 2023-12-02T03:11:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_11.status 2023-12-02T03:11:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-02T03:11:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:10.982 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=78 2023-12-02T03:11:10.982 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 139 = 78 2023-12-02T03:11:10.982 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:10.982 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T03:11:11.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_11.status 2023-12-02T03:11:11.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:11.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:11.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_11 2023-12-02T03:11:11.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_11.status 2023-12-02T03:11:11.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-02T03:11:11.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:11.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:11.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=78 2023-12-02T03:11:11.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 139 = 78 2023-12-02T03:11:11.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T03:11:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_11.status 2023-12-02T03:11:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_11 2023-12-02T03:11:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_11.status 2023-12-02T03:11:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-02T03:11:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:11:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:11:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:11:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:11:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:11:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 139 = '' 2023-12-02T03:11:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1.6 2023-12-02T03:11:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_11.status 2023-12-02T03:11:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_11 2023-12-02T03:11:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_11.status 2023-12-02T03:11:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-02T03:11:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:13.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:13.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=139 2023-12-02T03:11:13.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 139 = 139 2023-12-02T03:11:13.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:13.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2023-12-02T03:11:13.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:13.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:13.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_11 2023-12-02T03:11:13.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:11:13.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:11:13.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T03:11:13.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:11:14.401 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~184s 2023-12-02T03:11:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2023-12-02T03:11:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_11': 2023-12-02T03:11:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:11:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:11:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:11:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12115e9f77d7 2023-12-02T03:11:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12115e9f77d7 2023-12-02T03:11:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:11:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:11:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:11:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:11:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:01:38 2023 2023-12-02T03:11:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:01:38 2023 2023-12-02T03:11:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:01:38 2023 2023-12-02T03:11:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T03:11:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:11:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:11:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 258c1b03-48a5-42f2-8745-ac0283c8bc87 2023-12-02T03:11:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:11:15.005 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror image_11 snap 2023-12-02T03:11:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_11 2023-12-02T03:11:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:11:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T03:11:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_11.export 2023-12-02T03:11:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_11.export 2023-12-02T03:11:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_11.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_11.export 2023-12-02T03:11:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.kebs98RNzE/cluster2-mirror-image_11.export 2023-12-02T03:11:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:11:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.kebs98RNzE/cluster1-mirror-image_11.export 2023-12-02T03:11:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:11:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-image_11.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_11.export 2023-12-02T03:11:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_11.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_11.export 2023-12-02T03:11:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:11:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_12 2023-12-02T03:11:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror image_12 snap 2023-12-02T03:11:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_12 2023-12-02T03:11:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:11:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T03:11:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2023-12-02T03:11:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_12 2023-12-02T03:11:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2023-12-02T03:11:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_12 2023-12-02T03:11:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T03:11:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:11:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:11:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:11:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2023-12-02T03:11:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_12 2023-12-02T03:11:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T03:11:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T03:11:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T03:11:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2023-12-02T03:11:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T03:11:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T03:11:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T03:11:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T03:11:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T03:11:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:11:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T03:11:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:11:17.292 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:11:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:11:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2023-12-02T03:11:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:11:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:11:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:11:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:11:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:11:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:11:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:11:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:11:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:11:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:11:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:11:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:11:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:11:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:11:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:11:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:11:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:11:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:11:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:11:17.296 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:11:17.296 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:17.296 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:11:17.296 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:11:17.296 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:11:17.296 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:11:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:11:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:11:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:11:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:11:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:11:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:11:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2023-12-02T03:11:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-02T03:11:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:11:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:11:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:11:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2023-12-02T03:11:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:11:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:11:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:11:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:11:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:11:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:11:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:11:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:11:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:11:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:11:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:11:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:11:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:11:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:11:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:11:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:11:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:11:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:11:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:11:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:11:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:11:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:11:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:11:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:11:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:11:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:11:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:11:17.440 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:11:17.441 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:11:17.441 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:11:17.441 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2023-12-02T03:11:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-02T03:11:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:11:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:11:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:11:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_12 2023-12-02T03:11:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T03:11:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:11:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T03:11:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T03:11:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:11:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:11:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:11:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T03:11:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T03:11:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:11:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T03:11:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:11:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T03:11:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T03:11:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:11:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:11:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T03:11:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T03:11:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:11:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:11:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:11:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T03:11:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:11:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:11:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T03:11:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T03:11:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T03:11:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:11:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:11:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:11:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_12 2023-12-02T03:11:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-02T03:11:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:11:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:11:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:11:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_12 2023-12-02T03:11:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T03:11:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:11:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T03:11:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T03:11:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:11:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:11:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:11:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T03:11:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T03:11:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:11:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T03:11:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:11:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T03:11:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T03:11:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:11:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:11:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T03:11:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T03:11:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:11:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:11:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:11:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T03:11:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:11:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:11:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T03:11:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T03:11:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T03:11:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:11:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:11:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:11:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_12 2023-12-02T03:11:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T03:11:17.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T03:11:17.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T03:11:17.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T03:11:17.752 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T03:11:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T03:11:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T03:11:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2023-12-02T03:11:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:11:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_12 2023-12-02T03:11:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T03:11:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T03:11:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2023-12-02T03:11:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:11:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_12 2023-12-02T03:11:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-image_12.status 2023-12-02T03:11:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-image_12.status 2023-12-02T03:11:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:11:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_12.status 2023-12-02T03:11:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-image_12.status 2023-12-02T03:11:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-image_12.status 2023-12-02T03:11:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:11:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_12.status 2023-12-02T03:11:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2023-12-02T03:11:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_12 2023-12-02T03:11:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2023-12-02T03:11:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 141 2023-12-02T03:11:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.kebs98RNzE/cluster2-mirror-image_12.status 2023-12-02T03:11:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_12 2023-12-02T03:11:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_12.status 2023-12-02T03:11:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2023-12-02T03:11:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T03:11:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=141 2023-12-02T03:11:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:11:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T03:11:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_12.status 2023-12-02T03:11:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_12 2023-12-02T03:11:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_12.status 2023-12-02T03:11:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-02T03:11:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:11:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:11:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:11:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:11:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:11:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 141 = '' 2023-12-02T03:11:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T03:11:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_12.status 2023-12-02T03:11:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_12 2023-12-02T03:11:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_12.status 2023-12-02T03:11:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-02T03:11:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:11:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:11:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:11:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:11:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:11:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 141 = '' 2023-12-02T03:11:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T03:11:19.805 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~189s 2023-12-02T03:11:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_12.status 2023-12-02T03:11:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_12 2023-12-02T03:11:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_12.status 2023-12-02T03:11:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-02T03:11:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=141 2023-12-02T03:11:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 141 = 141 2023-12-02T03:11:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2023-12-02T03:11:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_12 2023-12-02T03:11:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:11:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:11:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T03:11:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:11:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2023-12-02T03:11:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_12': 2023-12-02T03:11:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:11:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:11:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:11:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12a110e3d821 2023-12-02T03:11:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12a110e3d821 2023-12-02T03:11:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:11:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:11:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:11:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:11:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:01:53 2023 2023-12-02T03:11:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:01:53 2023 2023-12-02T03:11:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:01:53 2023 2023-12-02T03:11:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T03:11:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:11:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:11:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: a30b9861-e056-4151-bcfe-e4a4f2e867dd 2023-12-02T03:11:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:11:21.020 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:21.021 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror image_12 snap 2023-12-02T03:11:21.021 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:21.021 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_12 2023-12-02T03:11:21.021 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:11:21.021 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T03:11:21.021 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_12.export 2023-12-02T03:11:21.021 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_12.export 2023-12-02T03:11:21.021 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_12.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_12.export 2023-12-02T03:11:21.021 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.kebs98RNzE/cluster2-mirror-image_12.export 2023-12-02T03:11:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:11:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.kebs98RNzE/cluster1-mirror-image_12.export 2023-12-02T03:11:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:11:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-image_12.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_12.export 2023-12-02T03:11:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_12.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_12.export 2023-12-02T03:11:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:11:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_13 2023-12-02T03:11:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror image_13 snap 2023-12-02T03:11:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_13 2023-12-02T03:11:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:11:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T03:11:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2023-12-02T03:11:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_13 2023-12-02T03:11:21.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2023-12-02T03:11:21.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:21.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:21.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_13 2023-12-02T03:11:21.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T03:11:21.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:11:21.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:11:21.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:11:22.801 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2023-12-02T03:11:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_13 2023-12-02T03:11:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T03:11:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T03:11:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T03:11:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2023-12-02T03:11:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T03:11:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T03:11:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T03:11:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T03:11:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T03:11:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:11:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T03:11:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:11:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:11:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:11:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2023-12-02T03:11:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:11:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:11:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:11:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:11:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:11:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:11:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:11:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:11:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:11:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:11:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:11:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:11:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:11:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:11:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:11:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:11:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:11:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:11:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:11:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:11:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:11:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:11:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:11:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:11:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:11:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:11:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:11:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:11:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:11:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:11:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2023-12-02T03:11:22.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2023-12-02T03:11:22.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:11:22.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:11:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:11:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_13 2023-12-02T03:11:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:11:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:11:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:11:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:11:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:11:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:11:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:11:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:11:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:11:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:11:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:11:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:11:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:11:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:11:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:11:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:11:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:11:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:11:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:11:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:11:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:11:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:11:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:11:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:11:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:11:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:11:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:11:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:11:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:11:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:11:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_13 2023-12-02T03:11:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T03:11:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T03:11:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T03:11:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T03:11:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T03:11:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T03:11:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T03:11:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2023-12-02T03:11:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:11:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_13 2023-12-02T03:11:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T03:11:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T03:11:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2023-12-02T03:11:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:11:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_13 2023-12-02T03:11:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-image_13.status 2023-12-02T03:11:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-image_13.status 2023-12-02T03:11:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:11:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_13.status 2023-12-02T03:11:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-image_13.status 2023-12-02T03:11:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-image_13.status 2023-12-02T03:11:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:11:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_13.status 2023-12-02T03:11:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2023-12-02T03:11:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_13 2023-12-02T03:11:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2023-12-02T03:11:23.812 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 143 2023-12-02T03:11:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.kebs98RNzE/cluster2-mirror-image_13.status 2023-12-02T03:11:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_13 2023-12-02T03:11:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_13.status 2023-12-02T03:11:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2023-12-02T03:11:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T03:11:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=143 2023-12-02T03:11:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:11:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T03:11:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_13.status 2023-12-02T03:11:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_13 2023-12-02T03:11:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_13.status 2023-12-02T03:11:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-02T03:11:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=80 2023-12-02T03:11:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 143 = 80 2023-12-02T03:11:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T03:11:24.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_13.status 2023-12-02T03:11:24.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:24.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:24.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_13 2023-12-02T03:11:24.529 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_13.status 2023-12-02T03:11:24.529 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-02T03:11:24.529 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:11:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:11:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:11:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:11:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:11:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 143 = '' 2023-12-02T03:11:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T03:11:25.209 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~195s 2023-12-02T03:11:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_13.status 2023-12-02T03:11:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_13 2023-12-02T03:11:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_13.status 2023-12-02T03:11:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-02T03:11:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:11:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:11:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:11:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:11:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:11:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 143 = '' 2023-12-02T03:11:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1.6 2023-12-02T03:11:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_13.status 2023-12-02T03:11:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_13 2023-12-02T03:11:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_13.status 2023-12-02T03:11:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-02T03:11:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=143 2023-12-02T03:11:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 143 = 143 2023-12-02T03:11:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2023-12-02T03:11:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_13 2023-12-02T03:11:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:11:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:11:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T03:11:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:11:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2023-12-02T03:11:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_13': 2023-12-02T03:11:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:11:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:11:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:11:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 10e49ec39909 2023-12-02T03:11:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.10e49ec39909 2023-12-02T03:11:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:11:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:11:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:11:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:11:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:02:03 2023 2023-12-02T03:11:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:02:03 2023 2023-12-02T03:11:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:02:03 2023 2023-12-02T03:11:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T03:11:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:11:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:11:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: cdf57a21-3ee2-466a-9e94-c4dd4500ddd1 2023-12-02T03:11:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:11:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror image_13 snap 2023-12-02T03:11:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_13 2023-12-02T03:11:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:11:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T03:11:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_13.export 2023-12-02T03:11:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_13.export 2023-12-02T03:11:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_13.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_13.export 2023-12-02T03:11:28.156 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.kebs98RNzE/cluster2-mirror-image_13.export 2023-12-02T03:11:28.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:11:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.kebs98RNzE/cluster1-mirror-image_13.export 2023-12-02T03:11:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:11:28.600 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-image_13.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_13.export 2023-12-02T03:11:28.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_13.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_13.export 2023-12-02T03:11:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:11:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_14 2023-12-02T03:11:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror image_14 snap 2023-12-02T03:11:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_14 2023-12-02T03:11:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:11:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T03:11:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2023-12-02T03:11:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_14 2023-12-02T03:11:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2023-12-02T03:11:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_14 2023-12-02T03:11:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T03:11:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:11:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:11:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:11:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2023-12-02T03:11:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_14 2023-12-02T03:11:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T03:11:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T03:11:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T03:11:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2023-12-02T03:11:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T03:11:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T03:11:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T03:11:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T03:11:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T03:11:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:11:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T03:11:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:11:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:11:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:11:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2023-12-02T03:11:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:11:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:11:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:11:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:11:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:11:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:11:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:11:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:11:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:11:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:11:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:11:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:11:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:11:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:11:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:11:30.286 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:11:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:11:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:11:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:11:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:11:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:11:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:11:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:11:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:11:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:11:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:11:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:11:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:11:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:11:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:11:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2023-12-02T03:11:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2023-12-02T03:11:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:11:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:11:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:11:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2023-12-02T03:11:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:11:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:11:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:11:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:11:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:11:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:11:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:11:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:11:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:11:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:11:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:11:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:11:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:11:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:11:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:11:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:11:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:11:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:11:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:11:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:11:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:11:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:11:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:11:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:11:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:11:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:11:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:11:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:11:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:11:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:11:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2023-12-02T03:11:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2023-12-02T03:11:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:11:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:11:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:11:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_14 2023-12-02T03:11:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T03:11:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:11:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T03:11:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T03:11:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:11:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:11:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:11:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T03:11:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T03:11:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:11:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T03:11:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:11:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T03:11:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T03:11:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:11:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:11:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T03:11:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T03:11:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:11:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:11:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:11:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T03:11:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:11:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:11:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T03:11:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T03:11:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T03:11:30.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:11:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:11:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:11:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_14 2023-12-02T03:11:30.612 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~200s 2023-12-02T03:11:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2023-12-02T03:11:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:11:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:11:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:11:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_14 2023-12-02T03:11:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T03:11:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:11:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T03:11:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T03:11:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:11:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:11:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:11:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T03:11:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T03:11:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:11:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T03:11:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:11:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T03:11:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T03:11:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:11:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:11:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T03:11:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T03:11:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:11:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:11:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:11:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T03:11:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:11:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:11:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T03:11:30.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T03:11:30.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T03:11:30.643 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:11:30.643 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:11:30.643 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:11:30.643 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_14 2023-12-02T03:11:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T03:11:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T03:11:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T03:11:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T03:11:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T03:11:30.738 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T03:11:30.738 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T03:11:30.738 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:30.738 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2023-12-02T03:11:30.738 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:11:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_14 2023-12-02T03:11:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T03:11:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T03:11:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2023-12-02T03:11:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:11:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_14 2023-12-02T03:11:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-image_14.status 2023-12-02T03:11:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-image_14.status 2023-12-02T03:11:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:11:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_14.status 2023-12-02T03:11:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-image_14.status 2023-12-02T03:11:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-image_14.status 2023-12-02T03:11:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:11:30.741 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_14.status 2023-12-02T03:11:30.741 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2023-12-02T03:11:30.741 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:30.741 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:30.741 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_14 2023-12-02T03:11:30.741 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2023-12-02T03:11:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 145 2023-12-02T03:11:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.kebs98RNzE/cluster2-mirror-image_14.status 2023-12-02T03:11:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_14 2023-12-02T03:11:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_14.status 2023-12-02T03:11:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2023-12-02T03:11:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T03:11:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=145 2023-12-02T03:11:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:11:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T03:11:31.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_14.status 2023-12-02T03:11:31.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:31.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:31.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_14 2023-12-02T03:11:31.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_14.status 2023-12-02T03:11:31.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-02T03:11:31.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:31.986 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=81 2023-12-02T03:11:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 145 = 81 2023-12-02T03:11:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T03:11:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_14.status 2023-12-02T03:11:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_14 2023-12-02T03:11:32.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_14.status 2023-12-02T03:11:32.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-02T03:11:32.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:32.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=81 2023-12-02T03:11:32.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 145 = 81 2023-12-02T03:11:32.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:32.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T03:11:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_14.status 2023-12-02T03:11:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_14 2023-12-02T03:11:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_14.status 2023-12-02T03:11:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-02T03:11:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:11:33.299 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:33.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:11:33.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:11:33.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:11:33.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:11:33.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 145 = '' 2023-12-02T03:11:33.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:33.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1.6 2023-12-02T03:11:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_14.status 2023-12-02T03:11:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_14 2023-12-02T03:11:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_14.status 2023-12-02T03:11:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-02T03:11:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:34.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:34.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=145 2023-12-02T03:11:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 145 = 145 2023-12-02T03:11:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2023-12-02T03:11:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_14 2023-12-02T03:11:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:11:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:11:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T03:11:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:11:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2023-12-02T03:11:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_14': 2023-12-02T03:11:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:11:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:11:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:11:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 11b838851b3 2023-12-02T03:11:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.11b838851b3 2023-12-02T03:11:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:11:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:11:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:11:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:11:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:02:16 2023 2023-12-02T03:11:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:02:16 2023 2023-12-02T03:11:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:02:16 2023 2023-12-02T03:11:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T03:11:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:11:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:11:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 1d600ef5-05fa-4ec8-bb36-7d891860097c 2023-12-02T03:11:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:11:36.015 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~205s 2023-12-02T03:11:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror image_14 snap 2023-12-02T03:11:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_14 2023-12-02T03:11:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:11:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T03:11:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_14.export 2023-12-02T03:11:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_14.export 2023-12-02T03:11:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_14.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_14.export 2023-12-02T03:11:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.kebs98RNzE/cluster2-mirror-image_14.export 2023-12-02T03:11:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:11:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.kebs98RNzE/cluster1-mirror-image_14.export 2023-12-02T03:11:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:11:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-image_14.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_14.export 2023-12-02T03:11:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_14.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_14.export 2023-12-02T03:11:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:11:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_15 2023-12-02T03:11:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror image_15 snap 2023-12-02T03:11:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_15 2023-12-02T03:11:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:11:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T03:11:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2023-12-02T03:11:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_15 2023-12-02T03:11:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2023-12-02T03:11:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_15 2023-12-02T03:11:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T03:11:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:11:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:11:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:11:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2023-12-02T03:11:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_15 2023-12-02T03:11:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T03:11:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T03:11:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T03:11:38.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2023-12-02T03:11:38.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T03:11:38.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T03:11:38.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:38.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T03:11:38.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T03:11:38.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T03:11:38.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:11:38.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T03:11:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:11:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:11:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:11:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2023-12-02T03:11:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:11:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:11:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:11:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:11:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:11:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:11:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:11:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:11:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:11:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:11:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:11:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:11:38.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:11:38.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:11:38.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:11:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:11:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:11:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:11:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:11:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:11:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:11:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:11:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:11:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:11:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:11:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:11:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:11:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:11:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:11:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:11:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2023-12-02T03:11:38.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T03:11:38.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T03:11:38.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T03:11:38.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T03:11:38.469 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T03:11:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T03:11:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T03:11:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2023-12-02T03:11:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:11:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_15 2023-12-02T03:11:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T03:11:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T03:11:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2023-12-02T03:11:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:11:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_15 2023-12-02T03:11:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-image_15.status 2023-12-02T03:11:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-image_15.status 2023-12-02T03:11:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:11:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_15.status 2023-12-02T03:11:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-image_15.status 2023-12-02T03:11:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-image_15.status 2023-12-02T03:11:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:11:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_15.status 2023-12-02T03:11:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2023-12-02T03:11:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_15 2023-12-02T03:11:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2023-12-02T03:11:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 147 2023-12-02T03:11:39.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.kebs98RNzE/cluster2-mirror-image_15.status 2023-12-02T03:11:39.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:39.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:39.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_15 2023-12-02T03:11:39.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_15.status 2023-12-02T03:11:39.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2023-12-02T03:11:39.371 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:39.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T03:11:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=147 2023-12-02T03:11:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:11:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T03:11:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_15.status 2023-12-02T03:11:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_15 2023-12-02T03:11:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_15.status 2023-12-02T03:11:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-02T03:11:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:11:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:11:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:11:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:11:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:11:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 147 = '' 2023-12-02T03:11:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T03:11:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_15.status 2023-12-02T03:11:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_15 2023-12-02T03:11:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_15.status 2023-12-02T03:11:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-02T03:11:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:40.129 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:11:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:11:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:11:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:11:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:11:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 147 = '' 2023-12-02T03:11:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T03:11:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_15.status 2023-12-02T03:11:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_15 2023-12-02T03:11:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_15.status 2023-12-02T03:11:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-02T03:11:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=147 2023-12-02T03:11:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 147 = 147 2023-12-02T03:11:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2023-12-02T03:11:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_15 2023-12-02T03:11:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:11:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:11:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T03:11:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:11:41.159 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7936236338902625) > weight (0.25) 2023-12-02T03:11:41.159 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.31971706510325215) > weight (0.25) 2023-12-02T03:11:41.159 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-02T03:11:41.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-02T03:11:41.160 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-02T03:11:41.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi057.stderr:2023-12-02T03:11:41.159+0000 7f62c50d8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 105409) UID: 1000 2023-12-02T03:11:41.419 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~211s 2023-12-02T03:11:41.982 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2023-12-02T03:11:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_15': 2023-12-02T03:11:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:11:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:11:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:11:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 1211da7dd79c 2023-12-02T03:11:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.1211da7dd79c 2023-12-02T03:11:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:11:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:11:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:11:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:11:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:02:26 2023 2023-12-02T03:11:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:02:26 2023 2023-12-02T03:11:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:02:26 2023 2023-12-02T03:11:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T03:11:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:11:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:11:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 5a34854e-65bf-43d7-9d2c-e0481cc5a159 2023-12-02T03:11:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:11:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror image_15 snap 2023-12-02T03:11:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_15 2023-12-02T03:11:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:11:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T03:11:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_15.export 2023-12-02T03:11:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_15.export 2023-12-02T03:11:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_15.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_15.export 2023-12-02T03:11:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.kebs98RNzE/cluster2-mirror-image_15.export 2023-12-02T03:11:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:11:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.kebs98RNzE/cluster1-mirror-image_15.export 2023-12-02T03:11:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:11:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-image_15.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_15.export 2023-12-02T03:11:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_15.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_15.export 2023-12-02T03:11:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:11:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_16 2023-12-02T03:11:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror image_16 snap 2023-12-02T03:11:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_16 2023-12-02T03:11:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:11:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T03:11:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2023-12-02T03:11:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_16 2023-12-02T03:11:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2023-12-02T03:11:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_16 2023-12-02T03:11:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T03:11:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:11:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:11:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:11:44.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2023-12-02T03:11:44.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:44.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:44.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_16 2023-12-02T03:11:44.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T03:11:44.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T03:11:44.380 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T03:11:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2023-12-02T03:11:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T03:11:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T03:11:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T03:11:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T03:11:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T03:11:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:11:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T03:11:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:11:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:11:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:11:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2023-12-02T03:11:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:11:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:11:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:11:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:11:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:11:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:11:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:11:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:11:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:11:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:11:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:11:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:11:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:11:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:11:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:11:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:11:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:11:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:11:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:11:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:11:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:11:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:11:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:11:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:11:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:11:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:11:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:11:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:11:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:11:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:11:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2023-12-02T03:11:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2023-12-02T03:11:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:11:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:11:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:11:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2023-12-02T03:11:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:11:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:11:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:11:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:11:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:11:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:11:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:11:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:11:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:11:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:11:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:11:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:11:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:11:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:11:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:11:44.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:11:44.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:11:44.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:11:44.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:11:44.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:11:44.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:44.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:11:44.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:11:44.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:11:44.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:11:44.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:11:44.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:11:44.502 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:11:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:11:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:11:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:11:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2023-12-02T03:11:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T03:11:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T03:11:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T03:11:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T03:11:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T03:11:44.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T03:11:44.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T03:11:44.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:44.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2023-12-02T03:11:44.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:11:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_16 2023-12-02T03:11:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T03:11:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T03:11:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2023-12-02T03:11:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:11:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_16 2023-12-02T03:11:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-image_16.status 2023-12-02T03:11:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:11:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-image_16.status 2023-12-02T03:11:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_16.status 2023-12-02T03:11:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-image_16.status 2023-12-02T03:11:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-image_16.status 2023-12-02T03:11:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:11:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_16.status 2023-12-02T03:11:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2023-12-02T03:11:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_16 2023-12-02T03:11:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2023-12-02T03:11:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 149 2023-12-02T03:11:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.kebs98RNzE/cluster2-mirror-image_16.status 2023-12-02T03:11:45.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:45.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:45.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_16 2023-12-02T03:11:45.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_16.status 2023-12-02T03:11:45.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2023-12-02T03:11:45.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T03:11:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=149 2023-12-02T03:11:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:11:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T03:11:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_16.status 2023-12-02T03:11:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_16 2023-12-02T03:11:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_16.status 2023-12-02T03:11:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-02T03:11:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=83 2023-12-02T03:11:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 149 = 83 2023-12-02T03:11:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T03:11:46.160 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-02T03:11:46.160 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T03:11:46.160 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-02T03:11:46.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-02T03:11:46.161 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-02T03:11:46.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-02T03:11:46.164 INFO:tasks.rbd_mirror_thrash:waiting for 92 secs before thrashing 2023-12-02T03:11:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_16.status 2023-12-02T03:11:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_16 2023-12-02T03:11:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_16.status 2023-12-02T03:11:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-02T03:11:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=83 2023-12-02T03:11:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 149 = 83 2023-12-02T03:11:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T03:11:46.823 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~216s 2023-12-02T03:11:47.240 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_16.status 2023-12-02T03:11:47.240 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_16 2023-12-02T03:11:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_16.status 2023-12-02T03:11:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-02T03:11:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:11:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:47.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:11:47.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:11:47.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:11:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:11:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 149 = '' 2023-12-02T03:11:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1.6 2023-12-02T03:11:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_16.status 2023-12-02T03:11:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_16 2023-12-02T03:11:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_16.status 2023-12-02T03:11:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-02T03:11:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:48.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=149 2023-12-02T03:11:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 149 = 149 2023-12-02T03:11:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2023-12-02T03:11:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_16 2023-12-02T03:11:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:11:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:11:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T03:11:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:11:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2023-12-02T03:11:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_16': 2023-12-02T03:11:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:11:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:11:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:11:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12b1e1c6312a 2023-12-02T03:11:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12b1e1c6312a 2023-12-02T03:11:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:11:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:11:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:11:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:11:50.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:02:37 2023 2023-12-02T03:11:50.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:02:37 2023 2023-12-02T03:11:50.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:02:37 2023 2023-12-02T03:11:50.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T03:11:50.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:11:50.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:11:50.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 807f0ce9-67ef-4e3a-987e-bfe34f97a832 2023-12-02T03:11:50.002 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:11:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror image_16 snap 2023-12-02T03:11:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_16 2023-12-02T03:11:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:11:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T03:11:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_16.export 2023-12-02T03:11:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_16.export 2023-12-02T03:11:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_16.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_16.export 2023-12-02T03:11:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.kebs98RNzE/cluster2-mirror-image_16.export 2023-12-02T03:11:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:11:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.kebs98RNzE/cluster1-mirror-image_16.export 2023-12-02T03:11:50.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:11:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-image_16.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_16.export 2023-12-02T03:11:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_16.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_16.export 2023-12-02T03:11:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:11:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_17 2023-12-02T03:11:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror image_17 snap 2023-12-02T03:11:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_17 2023-12-02T03:11:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:11:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T03:11:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2023-12-02T03:11:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_17 2023-12-02T03:11:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2023-12-02T03:11:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_17 2023-12-02T03:11:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T03:11:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:11:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:11:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:11:52.227 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~222s 2023-12-02T03:11:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2023-12-02T03:11:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_17 2023-12-02T03:11:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T03:11:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T03:11:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T03:11:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2023-12-02T03:11:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T03:11:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T03:11:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T03:11:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T03:11:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T03:11:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:11:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T03:11:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:11:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:11:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:11:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2023-12-02T03:11:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:11:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:11:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:11:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:11:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:11:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:11:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:11:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:11:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:11:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:11:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:11:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:11:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:11:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:11:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:11:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:11:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:11:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:11:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:11:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:11:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:11:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:11:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:11:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:11:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:11:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:11:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:11:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:11:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:11:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:11:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2023-12-02T03:11:52.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T03:11:52.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T03:11:52.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T03:11:52.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T03:11:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T03:11:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T03:11:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T03:11:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2023-12-02T03:11:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:11:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_17 2023-12-02T03:11:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T03:11:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T03:11:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2023-12-02T03:11:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:11:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:52.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:52.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_17 2023-12-02T03:11:52.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-image_17.status 2023-12-02T03:11:52.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-image_17.status 2023-12-02T03:11:52.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:11:52.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_17.status 2023-12-02T03:11:52.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-image_17.status 2023-12-02T03:11:52.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-image_17.status 2023-12-02T03:11:52.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:11:52.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_17.status 2023-12-02T03:11:52.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2023-12-02T03:11:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_17 2023-12-02T03:11:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2023-12-02T03:11:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 151 2023-12-02T03:11:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.kebs98RNzE/cluster2-mirror-image_17.status 2023-12-02T03:11:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_17 2023-12-02T03:11:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_17.status 2023-12-02T03:11:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2023-12-02T03:11:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:53.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T03:11:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=151 2023-12-02T03:11:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:11:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T03:11:53.593 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_17.status 2023-12-02T03:11:53.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:53.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:53.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_17 2023-12-02T03:11:53.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_17.status 2023-12-02T03:11:53.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-02T03:11:53.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:11:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:11:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:11:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:11:53.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:11:53.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 151 = '' 2023-12-02T03:11:53.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:53.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T03:11:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_17.status 2023-12-02T03:11:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_17 2023-12-02T03:11:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_17.status 2023-12-02T03:11:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-02T03:11:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:54.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:11:54.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:54.302 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:11:54.302 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:11:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:11:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:11:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 151 = '' 2023-12-02T03:11:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T03:11:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_17.status 2023-12-02T03:11:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_17 2023-12-02T03:11:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_17.status 2023-12-02T03:11:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-02T03:11:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=151 2023-12-02T03:11:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 151 = 151 2023-12-02T03:11:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2023-12-02T03:11:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_17 2023-12-02T03:11:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:11:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:11:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T03:11:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:11:56.357 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2023-12-02T03:11:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_17': 2023-12-02T03:11:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:11:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:11:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:11:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12b1cd66238 2023-12-02T03:11:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12b1cd66238 2023-12-02T03:11:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:11:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:11:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:11:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:11:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:02:51 2023 2023-12-02T03:11:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:02:51 2023 2023-12-02T03:11:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:02:51 2023 2023-12-02T03:11:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T03:11:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:11:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:11:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 1f4ea1ff-7710-42b2-9726-194a70ed8f1e 2023-12-02T03:11:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:11:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:56.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror image_17 snap 2023-12-02T03:11:56.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:56.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_17 2023-12-02T03:11:56.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:11:56.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T03:11:56.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_17.export 2023-12-02T03:11:56.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_17.export 2023-12-02T03:11:56.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_17.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_17.export 2023-12-02T03:11:56.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.kebs98RNzE/cluster2-mirror-image_17.export 2023-12-02T03:11:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:11:56.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.kebs98RNzE/cluster1-mirror-image_17.export 2023-12-02T03:11:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:11:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-image_17.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_17.export 2023-12-02T03:11:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_17.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_17.export 2023-12-02T03:11:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:11:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_18 2023-12-02T03:11:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror image_18 snap 2023-12-02T03:11:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_18 2023-12-02T03:11:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:11:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T03:11:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2023-12-02T03:11:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_18 2023-12-02T03:11:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2023-12-02T03:11:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_18 2023-12-02T03:11:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T03:11:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:11:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:11:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:11:57.631 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~227s 2023-12-02T03:11:58.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2023-12-02T03:11:58.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:58.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:58.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_18 2023-12-02T03:11:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T03:11:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T03:11:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T03:11:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2023-12-02T03:11:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T03:11:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T03:11:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T03:11:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T03:11:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T03:11:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:11:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T03:11:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:11:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:11:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:11:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2023-12-02T03:11:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:11:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:11:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:11:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:11:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:11:58.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:11:58.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:11:58.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:11:58.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:11:58.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:11:58.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:58.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:11:58.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:11:58.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:11:58.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:11:58.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:11:58.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:11:58.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:11:58.356 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:11:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:11:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:11:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:11:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:11:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:11:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:11:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:11:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:11:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:11:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:11:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:11:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:11:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2023-12-02T03:11:58.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2023-12-02T03:11:58.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:11:58.468 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:11:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:11:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2023-12-02T03:11:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:11:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:11:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:11:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:11:58.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:11:58.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:11:58.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:11:58.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:11:58.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:11:58.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:11:58.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:58.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:11:58.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:11:58.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:11:58.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:11:58.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:11:58.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:11:58.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:11:58.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:11:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:11:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:11:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:11:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:11:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:11:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:11:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:11:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:11:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:11:58.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:11:58.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:11:58.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:11:58.490 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2023-12-02T03:11:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2023-12-02T03:11:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:11:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:11:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:11:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_18 2023-12-02T03:11:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T03:11:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:11:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T03:11:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T03:11:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:11:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:11:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:11:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T03:11:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T03:11:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:11:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T03:11:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:11:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T03:11:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T03:11:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:11:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:11:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T03:11:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T03:11:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:11:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:11:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:11:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T03:11:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:11:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:11:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T03:11:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T03:11:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T03:11:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:11:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:11:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:11:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_18 2023-12-02T03:11:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2023-12-02T03:11:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:11:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:11:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:11:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_18 2023-12-02T03:11:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T03:11:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:11:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:11:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T03:11:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T03:11:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:11:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:11:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:11:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T03:11:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T03:11:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:11:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T03:11:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:11:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T03:11:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T03:11:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:11:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:11:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T03:11:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T03:11:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:11:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:11:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:11:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:11:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T03:11:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:11:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:11:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T03:11:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T03:11:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T03:11:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:11:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:11:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:11:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_18 2023-12-02T03:11:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T03:11:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T03:11:58.800 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T03:11:58.800 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T03:11:58.800 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T03:11:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T03:11:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T03:11:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:11:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2023-12-02T03:11:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:11:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_18 2023-12-02T03:11:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T03:11:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T03:11:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2023-12-02T03:11:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:11:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_18 2023-12-02T03:11:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-image_18.status 2023-12-02T03:11:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-image_18.status 2023-12-02T03:11:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:11:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_18.status 2023-12-02T03:11:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-image_18.status 2023-12-02T03:11:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-image_18.status 2023-12-02T03:11:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:11:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_18.status 2023-12-02T03:11:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2023-12-02T03:11:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_18 2023-12-02T03:11:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2023-12-02T03:11:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 153 2023-12-02T03:11:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.kebs98RNzE/cluster2-mirror-image_18.status 2023-12-02T03:11:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:11:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_18 2023-12-02T03:11:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_18.status 2023-12-02T03:11:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2023-12-02T03:11:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T03:11:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=153 2023-12-02T03:11:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:11:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T03:11:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_18.status 2023-12-02T03:11:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:11:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:11:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_18 2023-12-02T03:11:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_18.status 2023-12-02T03:11:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-02T03:11:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:11:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:11:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:11:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:11:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:11:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:11:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:11:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 153 = '' 2023-12-02T03:11:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:11:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T03:12:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_18.status 2023-12-02T03:12:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_18 2023-12-02T03:12:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_18.status 2023-12-02T03:12:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-02T03:12:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:12:00.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:12:00.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:12:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:12:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:12:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:12:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:12:00.134 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 153 = '' 2023-12-02T03:12:00.134 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:12:00.134 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T03:12:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_18.status 2023-12-02T03:12:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_18 2023-12-02T03:12:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_18.status 2023-12-02T03:12:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-02T03:12:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:12:00.981 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:12:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=153 2023-12-02T03:12:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 153 = 153 2023-12-02T03:12:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:12:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2023-12-02T03:12:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_18 2023-12-02T03:12:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:12:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:12:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T03:12:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:12:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2023-12-02T03:12:02.032 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_18': 2023-12-02T03:12:02.032 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:12:02.032 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:12:02.032 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:12:02.032 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12118d51a868 2023-12-02T03:12:02.032 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12118d51a868 2023-12-02T03:12:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:12:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:12:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:12:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:12:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:03:00 2023 2023-12-02T03:12:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:03:00 2023 2023-12-02T03:12:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:03:00 2023 2023-12-02T03:12:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T03:12:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:12:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:12:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: b8d1a79d-fa19-4b64-ba6b-5a4658335ba3 2023-12-02T03:12:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:12:02.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:12:02.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror image_18 snap 2023-12-02T03:12:02.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:02.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_18 2023-12-02T03:12:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:12:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T03:12:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_18.export 2023-12-02T03:12:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_18.export 2023-12-02T03:12:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_18.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_18.export 2023-12-02T03:12:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.kebs98RNzE/cluster2-mirror-image_18.export 2023-12-02T03:12:02.211 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:12:02.218 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.kebs98RNzE/cluster1-mirror-image_18.export 2023-12-02T03:12:02.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:12:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-image_18.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_18.export 2023-12-02T03:12:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_18.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_18.export 2023-12-02T03:12:02.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:12:02.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:12:02.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_19 2023-12-02T03:12:02.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror image_19 snap 2023-12-02T03:12:02.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:12:02.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:02.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_19 2023-12-02T03:12:02.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:12:02.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T03:12:03.035 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~232s 2023-12-02T03:12:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2023-12-02T03:12:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_19 2023-12-02T03:12:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2023-12-02T03:12:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_19 2023-12-02T03:12:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T03:12:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:12:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:12:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:12:04.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2023-12-02T03:12:04.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:04.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:04.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_19 2023-12-02T03:12:04.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T03:12:04.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T03:12:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T03:12:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2023-12-02T03:12:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T03:12:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T03:12:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T03:12:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T03:12:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T03:12:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:12:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T03:12:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:12:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:12:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2023-12-02T03:12:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:12:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:12:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:12:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:12:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:12:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:12:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:12:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:12:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:12:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:12:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:12:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:12:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:12:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:12:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:12:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:12:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2023-12-02T03:12:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2023-12-02T03:12:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:12:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:12:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_19 2023-12-02T03:12:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:12:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:12:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:12:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:12:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:12:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:12:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:12:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:12:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:12:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:12:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:12:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:12:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:12:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:12:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:12:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:12:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:12:04.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:12:04.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:12:04.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_19 2023-12-02T03:12:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T03:12:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T03:12:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T03:12:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T03:12:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T03:12:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T03:12:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T03:12:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:12:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2023-12-02T03:12:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:12:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:12:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_19 2023-12-02T03:12:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T03:12:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T03:12:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2023-12-02T03:12:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:12:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:12:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_19 2023-12-02T03:12:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-image_19.status 2023-12-02T03:12:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-image_19.status 2023-12-02T03:12:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:12:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_19.status 2023-12-02T03:12:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-image_19.status 2023-12-02T03:12:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-image_19.status 2023-12-02T03:12:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:12:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_19.status 2023-12-02T03:12:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2023-12-02T03:12:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:12:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_19 2023-12-02T03:12:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2023-12-02T03:12:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 155 2023-12-02T03:12:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.kebs98RNzE/cluster2-mirror-image_19.status 2023-12-02T03:12:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:12:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_19 2023-12-02T03:12:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_19.status 2023-12-02T03:12:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2023-12-02T03:12:05.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:12:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T03:12:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=155 2023-12-02T03:12:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:12:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:12:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T03:12:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_19.status 2023-12-02T03:12:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_19 2023-12-02T03:12:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_19.status 2023-12-02T03:12:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-02T03:12:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:12:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:12:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=86 2023-12-02T03:12:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 155 = 86 2023-12-02T03:12:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:12:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T03:12:06.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_19.status 2023-12-02T03:12:06.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:06.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_19 2023-12-02T03:12:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_19.status 2023-12-02T03:12:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-02T03:12:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:12:06.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:12:06.461 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=86 2023-12-02T03:12:06.461 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 155 = 86 2023-12-02T03:12:06.461 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:12:06.461 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T03:12:07.263 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_19.status 2023-12-02T03:12:07.263 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:07.263 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:07.263 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_19 2023-12-02T03:12:07.263 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_19.status 2023-12-02T03:12:07.263 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-02T03:12:07.263 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:12:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:12:07.318 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:12:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:12:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:12:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:12:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:12:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 155 = '' 2023-12-02T03:12:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:12:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1.6 2023-12-02T03:12:08.439 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~238s 2023-12-02T03:12:08.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_19.status 2023-12-02T03:12:08.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:08.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:08.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_19 2023-12-02T03:12:08.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_19.status 2023-12-02T03:12:08.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-02T03:12:08.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:12:08.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:12:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=155 2023-12-02T03:12:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 155 = 155 2023-12-02T03:12:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:12:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2023-12-02T03:12:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_19 2023-12-02T03:12:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:12:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:12:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T03:12:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:12:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2023-12-02T03:12:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_19': 2023-12-02T03:12:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:12:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:12:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:12:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12a1f60a4dad 2023-12-02T03:12:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12a1f60a4dad 2023-12-02T03:12:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:12:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:12:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:12:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:12:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:03:13 2023 2023-12-02T03:12:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:03:13 2023 2023-12-02T03:12:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:03:13 2023 2023-12-02T03:12:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T03:12:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:12:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:12:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: faedaa7c-e337-491d-80ec-3cfa45a9493c 2023-12-02T03:12:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:12:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:12:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror image_19 snap 2023-12-02T03:12:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_19 2023-12-02T03:12:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:12:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T03:12:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_19.export 2023-12-02T03:12:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_19.export 2023-12-02T03:12:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_19.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_19.export 2023-12-02T03:12:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.kebs98RNzE/cluster2-mirror-image_19.export 2023-12-02T03:12:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:12:10.221 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.kebs98RNzE/cluster1-mirror-image_19.export 2023-12-02T03:12:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:12:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-image_19.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_19.export 2023-12-02T03:12:10.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_19.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_19.export 2023-12-02T03:12:10.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:12:10.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:12:10.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_20 2023-12-02T03:12:10.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror image_20 snap 2023-12-02T03:12:10.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:12:10.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:10.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_20 2023-12-02T03:12:10.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:12:10.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T03:12:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2023-12-02T03:12:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_20 2023-12-02T03:12:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2023-12-02T03:12:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_20 2023-12-02T03:12:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T03:12:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:12:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:12:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:12:11.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2023-12-02T03:12:11.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:11.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:11.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_20 2023-12-02T03:12:11.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T03:12:11.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T03:12:11.670 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T03:12:11.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2023-12-02T03:12:11.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T03:12:11.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:11.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T03:12:11.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T03:12:11.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T03:12:11.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T03:12:11.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:12:11.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T03:12:11.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:12:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:12:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2023-12-02T03:12:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:12:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:12:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:12:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:12:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:12:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:12:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:12:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:12:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:12:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:12:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:12:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:12:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:12:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:12:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:12:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:12:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2023-12-02T03:12:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-02T03:12:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:12:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:12:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2023-12-02T03:12:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:12:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:12:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:12:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:12:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:12:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:12:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:12:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:12:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:12:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:12:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:12:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:12:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:12:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:12:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:12:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:12:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:12:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:12:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:12:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2023-12-02T03:12:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T03:12:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T03:12:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T03:12:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T03:12:11.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T03:12:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T03:12:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T03:12:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:12:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2023-12-02T03:12:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:12:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:12:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_20 2023-12-02T03:12:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T03:12:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T03:12:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2023-12-02T03:12:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:12:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:12:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_20 2023-12-02T03:12:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-image_20.status 2023-12-02T03:12:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-image_20.status 2023-12-02T03:12:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:12:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_20.status 2023-12-02T03:12:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-image_20.status 2023-12-02T03:12:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-image_20.status 2023-12-02T03:12:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:12:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_20.status 2023-12-02T03:12:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2023-12-02T03:12:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:12:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_20 2023-12-02T03:12:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2023-12-02T03:12:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 157 2023-12-02T03:12:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.kebs98RNzE/cluster2-mirror-image_20.status 2023-12-02T03:12:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:12:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_20 2023-12-02T03:12:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_20.status 2023-12-02T03:12:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2023-12-02T03:12:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:12:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T03:12:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=157 2023-12-02T03:12:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:12:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:12:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T03:12:12.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_20.status 2023-12-02T03:12:12.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:12.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:12.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_20 2023-12-02T03:12:12.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_20.status 2023-12-02T03:12:12.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-02T03:12:12.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:12:12.986 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:12:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=87 2023-12-02T03:12:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 157 = 87 2023-12-02T03:12:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:12:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T03:12:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_20.status 2023-12-02T03:12:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_20 2023-12-02T03:12:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_20.status 2023-12-02T03:12:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-02T03:12:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:12:13.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:12:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=87 2023-12-02T03:12:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 157 = 87 2023-12-02T03:12:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:12:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T03:12:13.843 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~243s 2023-12-02T03:12:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_20.status 2023-12-02T03:12:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_20 2023-12-02T03:12:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_20.status 2023-12-02T03:12:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-02T03:12:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:12:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:12:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:12:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:12:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:12:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:12:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:12:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 157 = '' 2023-12-02T03:12:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:12:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1.6 2023-12-02T03:12:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_20.status 2023-12-02T03:12:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_20 2023-12-02T03:12:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_20.status 2023-12-02T03:12:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-02T03:12:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:12:15.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:12:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=157 2023-12-02T03:12:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 157 = 157 2023-12-02T03:12:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:12:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2023-12-02T03:12:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_20 2023-12-02T03:12:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:12:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:12:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T03:12:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:12:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2023-12-02T03:12:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_20': 2023-12-02T03:12:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:12:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:12:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:12:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12b1f96fe282 2023-12-02T03:12:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12b1f96fe282 2023-12-02T03:12:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:12:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:12:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:12:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:12:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:03:25 2023 2023-12-02T03:12:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:03:25 2023 2023-12-02T03:12:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:03:25 2023 2023-12-02T03:12:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T03:12:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:12:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:12:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 8117a00a-84c1-4b06-96bd-60840c20cce4 2023-12-02T03:12:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:12:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:12:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror image_20 snap 2023-12-02T03:12:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_20 2023-12-02T03:12:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:12:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T03:12:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_20.export 2023-12-02T03:12:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_20.export 2023-12-02T03:12:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_20.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_20.export 2023-12-02T03:12:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.kebs98RNzE/cluster2-mirror-image_20.export 2023-12-02T03:12:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:12:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.kebs98RNzE/cluster1-mirror-image_20.export 2023-12-02T03:12:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:12:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-image_20.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_20.export 2023-12-02T03:12:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_20.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_20.export 2023-12-02T03:12:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:12:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:12:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_21 2023-12-02T03:12:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror image_21 snap 2023-12-02T03:12:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:12:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_21 2023-12-02T03:12:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:12:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T03:12:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2023-12-02T03:12:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_21 2023-12-02T03:12:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2023-12-02T03:12:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_21 2023-12-02T03:12:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T03:12:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:12:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:12:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:12:19.247 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~249s 2023-12-02T03:12:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2023-12-02T03:12:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_21 2023-12-02T03:12:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T03:12:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T03:12:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T03:12:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2023-12-02T03:12:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T03:12:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T03:12:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T03:12:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T03:12:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T03:12:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:12:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T03:12:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:12:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:12:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2023-12-02T03:12:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:12:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:12:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:12:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:12:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:12:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:12:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:12:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:12:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:12:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:12:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:12:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:12:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:12:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:12:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:12:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:12:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2023-12-02T03:12:19.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T03:12:19.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T03:12:19.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T03:12:19.472 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T03:12:19.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T03:12:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T03:12:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T03:12:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:12:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2023-12-02T03:12:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:12:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:12:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_21 2023-12-02T03:12:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T03:12:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T03:12:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2023-12-02T03:12:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:12:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:12:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_21 2023-12-02T03:12:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-image_21.status 2023-12-02T03:12:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-image_21.status 2023-12-02T03:12:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:12:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_21.status 2023-12-02T03:12:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-image_21.status 2023-12-02T03:12:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-image_21.status 2023-12-02T03:12:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:12:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_21.status 2023-12-02T03:12:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2023-12-02T03:12:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:12:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_21 2023-12-02T03:12:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2023-12-02T03:12:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 159 2023-12-02T03:12:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.kebs98RNzE/cluster2-mirror-image_21.status 2023-12-02T03:12:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:12:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_21 2023-12-02T03:12:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_21.status 2023-12-02T03:12:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2023-12-02T03:12:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:12:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T03:12:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=159 2023-12-02T03:12:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:12:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:12:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T03:12:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_21.status 2023-12-02T03:12:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_21 2023-12-02T03:12:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_21.status 2023-12-02T03:12:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-02T03:12:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:12:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:12:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:12:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:12:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:12:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:12:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:12:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 159 = '' 2023-12-02T03:12:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:12:20.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T03:12:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_21.status 2023-12-02T03:12:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_21 2023-12-02T03:12:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_21.status 2023-12-02T03:12:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-02T03:12:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:12:21.323 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:12:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:12:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:12:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:12:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:12:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:12:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 159 = '' 2023-12-02T03:12:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:12:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T03:12:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_21.status 2023-12-02T03:12:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_21 2023-12-02T03:12:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_21.status 2023-12-02T03:12:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-02T03:12:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:12:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:12:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=159 2023-12-02T03:12:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 159 = 159 2023-12-02T03:12:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:12:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2023-12-02T03:12:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_21 2023-12-02T03:12:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:12:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:12:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T03:12:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:12:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2023-12-02T03:12:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_21': 2023-12-02T03:12:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:12:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:12:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:12:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 11b8936b5756 2023-12-02T03:12:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.11b8936b5756 2023-12-02T03:12:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:12:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:12:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:12:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:12:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:03:32 2023 2023-12-02T03:12:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:03:32 2023 2023-12-02T03:12:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:03:32 2023 2023-12-02T03:12:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T03:12:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:12:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:12:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: b629353c-89f1-46c0-b5ff-f652e93d5cb3 2023-12-02T03:12:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:12:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:12:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror image_21 snap 2023-12-02T03:12:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_21 2023-12-02T03:12:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:12:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T03:12:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_21.export 2023-12-02T03:12:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_21.export 2023-12-02T03:12:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_21.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_21.export 2023-12-02T03:12:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.kebs98RNzE/cluster2-mirror-image_21.export 2023-12-02T03:12:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:12:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.kebs98RNzE/cluster1-mirror-image_21.export 2023-12-02T03:12:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:12:23.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-image_21.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_21.export 2023-12-02T03:12:23.740 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_21.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_21.export 2023-12-02T03:12:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:12:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:12:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_22 2023-12-02T03:12:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror image_22 snap 2023-12-02T03:12:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:12:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_22 2023-12-02T03:12:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:12:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T03:12:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2023-12-02T03:12:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_22 2023-12-02T03:12:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2023-12-02T03:12:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_22 2023-12-02T03:12:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T03:12:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:12:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:12:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:12:24.650 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~254s 2023-12-02T03:12:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2023-12-02T03:12:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_22 2023-12-02T03:12:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T03:12:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T03:12:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T03:12:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2023-12-02T03:12:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T03:12:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T03:12:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T03:12:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T03:12:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T03:12:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:12:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T03:12:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:12:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:12:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2023-12-02T03:12:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:12:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:12:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:12:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:12:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:12:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:12:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:12:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:12:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:12:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:12:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:12:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:12:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:12:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:12:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:12:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:12:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2023-12-02T03:12:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2023-12-02T03:12:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:12:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:12:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2023-12-02T03:12:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:12:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:12:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:12:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:12:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:12:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:12:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:12:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:12:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:12:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:12:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:12:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:12:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:12:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:12:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:12:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:12:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:12:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:12:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:12:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2023-12-02T03:12:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T03:12:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T03:12:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T03:12:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T03:12:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T03:12:25.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T03:12:25.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T03:12:25.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:12:25.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2023-12-02T03:12:25.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:12:25.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:12:25.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:25.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_22 2023-12-02T03:12:25.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T03:12:25.604 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T03:12:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2023-12-02T03:12:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:12:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:12:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_22 2023-12-02T03:12:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-image_22.status 2023-12-02T03:12:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-image_22.status 2023-12-02T03:12:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:12:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_22.status 2023-12-02T03:12:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-image_22.status 2023-12-02T03:12:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-image_22.status 2023-12-02T03:12:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:12:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_22.status 2023-12-02T03:12:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2023-12-02T03:12:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:12:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_22 2023-12-02T03:12:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2023-12-02T03:12:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 161 2023-12-02T03:12:26.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.kebs98RNzE/cluster2-mirror-image_22.status 2023-12-02T03:12:26.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:12:26.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:26.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_22 2023-12-02T03:12:26.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_22.status 2023-12-02T03:12:26.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2023-12-02T03:12:26.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:12:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T03:12:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=161 2023-12-02T03:12:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:12:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:12:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T03:12:26.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_22.status 2023-12-02T03:12:26.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:26.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:26.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_22 2023-12-02T03:12:26.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_22.status 2023-12-02T03:12:26.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-02T03:12:26.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:12:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:12:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:12:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:12:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:12:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:12:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:12:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 161 = '' 2023-12-02T03:12:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:12:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T03:12:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_22.status 2023-12-02T03:12:27.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:27.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:27.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_22 2023-12-02T03:12:27.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_22.status 2023-12-02T03:12:27.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-02T03:12:27.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:12:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:12:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:12:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:12:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:12:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:12:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:12:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 161 = '' 2023-12-02T03:12:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:12:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T03:12:27.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_22.status 2023-12-02T03:12:27.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:27.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:27.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_22 2023-12-02T03:12:27.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_22.status 2023-12-02T03:12:27.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-02T03:12:27.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:12:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:12:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=161 2023-12-02T03:12:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 161 = 161 2023-12-02T03:12:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:12:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2023-12-02T03:12:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_22 2023-12-02T03:12:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:12:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:12:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T03:12:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:12:28.047 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-02T03:12:28.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-02T03:12:28.047 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-02T03:12:28.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi057.stderr:2023-12-02T03:12:28.047+0000 7fe24c8b1640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 106373) UID: 1000 2023-12-02T03:12:28.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2023-12-02T03:12:29.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_22': 2023-12-02T03:12:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:12:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:12:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:12:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12b1ed8e617c 2023-12-02T03:12:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12b1ed8e617c 2023-12-02T03:12:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:12:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:12:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:12:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:12:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:04:06 2023 2023-12-02T03:12:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:04:06 2023 2023-12-02T03:12:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:04:06 2023 2023-12-02T03:12:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T03:12:29.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:12:29.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:12:29.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 8ef4933a-3981-486a-8a90-10311d67bf00 2023-12-02T03:12:29.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:12:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:12:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror image_22 snap 2023-12-02T03:12:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_22 2023-12-02T03:12:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:12:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T03:12:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_22.export 2023-12-02T03:12:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_22.export 2023-12-02T03:12:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_22.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_22.export 2023-12-02T03:12:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.kebs98RNzE/cluster2-mirror-image_22.export 2023-12-02T03:12:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:12:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.kebs98RNzE/cluster1-mirror-image_22.export 2023-12-02T03:12:29.491 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:12:29.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-image_22.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_22.export 2023-12-02T03:12:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_22.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_22.export 2023-12-02T03:12:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:12:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:12:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_23 2023-12-02T03:12:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror image_23 snap 2023-12-02T03:12:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:12:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_23 2023-12-02T03:12:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:12:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T03:12:30.054 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~259s 2023-12-02T03:12:30.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2023-12-02T03:12:30.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:30.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:30.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_23 2023-12-02T03:12:30.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2023-12-02T03:12:30.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:30.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:30.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_23 2023-12-02T03:12:30.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T03:12:30.417 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:12:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:12:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:12:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2023-12-02T03:12:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_23 2023-12-02T03:12:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T03:12:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T03:12:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T03:12:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2023-12-02T03:12:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T03:12:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T03:12:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T03:12:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T03:12:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T03:12:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:12:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T03:12:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:12:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:12:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2023-12-02T03:12:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:12:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:12:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:12:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:12:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:12:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:12:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:12:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:12:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:12:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:12:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:12:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:12:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:12:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:12:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:12:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:12:31.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:31.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:31.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:31.444 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2023-12-02T03:12:31.517 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-02T03:12:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2023-12-02T03:12:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:12:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:12:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:12:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:12:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:12:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:12:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:12:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:12:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:12:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:12:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:12:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:12:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:12:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:12:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:12:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:12:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:12:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:12:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:12:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2023-12-02T03:12:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2023-12-02T03:12:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:12:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:12:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_23 2023-12-02T03:12:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T03:12:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T03:12:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T03:12:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T03:12:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T03:12:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T03:12:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:12:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T03:12:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T03:12:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T03:12:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T03:12:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:12:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T03:12:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T03:12:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T03:12:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T03:12:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:12:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:12:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:12:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_23 2023-12-02T03:12:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T03:12:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T03:12:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T03:12:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T03:12:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T03:12:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T03:12:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T03:12:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:12:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2023-12-02T03:12:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:12:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:12:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_23 2023-12-02T03:12:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T03:12:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T03:12:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2023-12-02T03:12:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:12:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:12:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_23 2023-12-02T03:12:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-image_23.status 2023-12-02T03:12:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-image_23.status 2023-12-02T03:12:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:12:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_23.status 2023-12-02T03:12:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-image_23.status 2023-12-02T03:12:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-image_23.status 2023-12-02T03:12:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:12:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_23.status 2023-12-02T03:12:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2023-12-02T03:12:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:12:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_23 2023-12-02T03:12:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2023-12-02T03:12:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 163 2023-12-02T03:12:32.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.kebs98RNzE/cluster2-mirror-image_23.status 2023-12-02T03:12:32.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:12:32.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:32.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_23 2023-12-02T03:12:32.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_23.status 2023-12-02T03:12:32.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2023-12-02T03:12:32.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:12:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T03:12:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=163 2023-12-02T03:12:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:12:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:12:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T03:12:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_23.status 2023-12-02T03:12:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_23 2023-12-02T03:12:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_23.status 2023-12-02T03:12:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-02T03:12:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:12:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:12:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:12:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:12:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:12:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:12:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:12:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 163 = '' 2023-12-02T03:12:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:12:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T03:12:33.139 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_23.status 2023-12-02T03:12:33.139 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:33.139 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:33.139 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_23 2023-12-02T03:12:33.139 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_23.status 2023-12-02T03:12:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-02T03:12:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:12:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:12:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:12:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:12:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:12:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:12:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:12:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 163 = '' 2023-12-02T03:12:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:12:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T03:12:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_23.status 2023-12-02T03:12:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_23 2023-12-02T03:12:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_23.status 2023-12-02T03:12:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-02T03:12:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:12:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:12:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=163 2023-12-02T03:12:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 163 = 163 2023-12-02T03:12:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:12:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2023-12-02T03:12:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_23 2023-12-02T03:12:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:12:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:12:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T03:12:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:12:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2023-12-02T03:12:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_23': 2023-12-02T03:12:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:12:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:12:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:12:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12c0b00e005 2023-12-02T03:12:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12c0b00e005 2023-12-02T03:12:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:12:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:12:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:12:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:12:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:03:54 2023 2023-12-02T03:12:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:03:54 2023 2023-12-02T03:12:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:03:54 2023 2023-12-02T03:12:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T03:12:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:12:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:12:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 8696705f-855b-402d-afd6-1c4d55e06c91 2023-12-02T03:12:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:12:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:12:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror image_23 snap 2023-12-02T03:12:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_23 2023-12-02T03:12:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:12:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T03:12:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_23.export 2023-12-02T03:12:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_23.export 2023-12-02T03:12:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_23.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_23.export 2023-12-02T03:12:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.kebs98RNzE/cluster2-mirror-image_23.export 2023-12-02T03:12:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:12:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.kebs98RNzE/cluster1-mirror-image_23.export 2023-12-02T03:12:35.458 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~265s 2023-12-02T03:12:35.479 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:12:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-image_23.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_23.export 2023-12-02T03:12:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_23.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_23.export 2023-12-02T03:12:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:12:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:12:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_24 2023-12-02T03:12:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror image_24 snap 2023-12-02T03:12:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:12:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_24 2023-12-02T03:12:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:12:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T03:12:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2023-12-02T03:12:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_24 2023-12-02T03:12:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2023-12-02T03:12:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_24 2023-12-02T03:12:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T03:12:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:12:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:12:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:12:37.048 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-02T03:12:37.048 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T03:12:37.048 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-02T03:12:37.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-02T03:12:37.048 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-02T03:12:37.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-02T03:12:37.051 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2023-12-02T03:12:37.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2023-12-02T03:12:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:12:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:12:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_24 2023-12-02T03:12:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T03:12:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T03:12:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T03:12:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2023-12-02T03:12:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T03:12:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T03:12:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T03:12:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T03:12:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T03:12:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:12:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T03:12:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:12:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:12:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2023-12-02T03:12:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:12:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:12:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:12:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:12:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:12:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:12:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:12:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:12:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:12:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:12:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:12:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:12:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:12:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:12:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:12:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:12:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2023-12-02T03:12:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-02T03:12:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:12:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:12:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2023-12-02T03:12:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:12:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:12:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:12:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:12:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:12:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:12:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:12:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:12:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:12:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:12:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:12:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:12:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:12:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:12:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:12:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:12:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:12:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:12:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:12:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2023-12-02T03:12:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-02T03:12:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:12:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:12:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_24 2023-12-02T03:12:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T03:12:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T03:12:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T03:12:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T03:12:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T03:12:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T03:12:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:12:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T03:12:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T03:12:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T03:12:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T03:12:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:12:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T03:12:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T03:12:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T03:12:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T03:12:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:12:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:12:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:12:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_24 2023-12-02T03:12:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-02T03:12:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:12:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:12:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_24 2023-12-02T03:12:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T03:12:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T03:12:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T03:12:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T03:12:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T03:12:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T03:12:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:12:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T03:12:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T03:12:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T03:12:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T03:12:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:12:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T03:12:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T03:12:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T03:12:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T03:12:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:12:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:12:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:12:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_24 2023-12-02T03:12:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-02T03:12:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:12:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:12:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:12:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 1 2023-12-02T03:12:38.858 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:12:38.858 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:12:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2023-12-02T03:12:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:12:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:12:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:12:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:12:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:12:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:12:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:12:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:12:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:12:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:12:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:12:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:12:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:12:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:12:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:12:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:12:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2023-12-02T03:12:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-02T03:12:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:12:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:12:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2023-12-02T03:12:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:12:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:12:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:12:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:12:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:12:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:12:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:12:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:12:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:12:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:12:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:12:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:12:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:12:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:12:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:12:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:12:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:12:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:12:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:12:38.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2023-12-02T03:12:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-02T03:12:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:12:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:12:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_24 2023-12-02T03:12:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T03:12:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T03:12:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T03:12:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T03:12:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T03:12:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T03:12:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:12:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T03:12:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T03:12:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T03:12:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T03:12:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:12:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T03:12:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T03:12:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T03:12:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T03:12:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:12:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:12:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:12:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_24 2023-12-02T03:12:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-02T03:12:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:12:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:12:39.186 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:39.186 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_24 2023-12-02T03:12:39.186 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T03:12:39.186 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:39.186 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:39.186 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T03:12:39.186 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T03:12:39.186 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:39.186 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T03:12:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T03:12:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T03:12:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:12:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T03:12:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T03:12:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T03:12:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T03:12:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:12:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T03:12:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T03:12:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T03:12:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T03:12:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:12:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:12:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:12:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_24 2023-12-02T03:12:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-02T03:12:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:12:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:12:39.292 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:12:39.292 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 2 2023-12-02T03:12:40.861 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~270s 2023-12-02T03:12:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:12:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:12:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2023-12-02T03:12:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:12:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:12:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:12:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:12:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:12:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:12:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:12:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:12:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:12:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:12:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:12:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:12:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:12:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:12:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:12:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:12:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2023-12-02T03:12:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-02T03:12:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:12:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:12:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2023-12-02T03:12:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:12:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:12:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:12:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:12:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:12:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:12:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:12:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:12:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:12:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:12:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:12:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:12:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:12:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:12:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:12:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:12:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:12:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:12:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:12:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2023-12-02T03:12:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-02T03:12:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:12:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:12:41.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:41.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_24 2023-12-02T03:12:41.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T03:12:41.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:41.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:41.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T03:12:41.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T03:12:41.513 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T03:12:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T03:12:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T03:12:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:12:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T03:12:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T03:12:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T03:12:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T03:12:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:12:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T03:12:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T03:12:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T03:12:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T03:12:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:12:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:12:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:12:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_24 2023-12-02T03:12:41.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-02T03:12:41.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:12:41.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:12:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_24 2023-12-02T03:12:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T03:12:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T03:12:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T03:12:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T03:12:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T03:12:41.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:41.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:41.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T03:12:41.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:12:41.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T03:12:41.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T03:12:41.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:41.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:41.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T03:12:41.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T03:12:41.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:41.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:41.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:41.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:12:41.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T03:12:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T03:12:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T03:12:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T03:12:41.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:12:41.636 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:12:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:12:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_24 2023-12-02T03:12:41.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-02T03:12:41.720 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:12:41.720 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:12:41.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:12:41.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 4 2023-12-02T03:12:45.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:12:45.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:12:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2023-12-02T03:12:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:12:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:12:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:12:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:12:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:12:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:12:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:12:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:12:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:12:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:12:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:12:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:12:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:12:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:12:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:12:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:12:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2023-12-02T03:12:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-02T03:12:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:12:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:12:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2023-12-02T03:12:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:12:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:12:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:12:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:12:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:12:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:12:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:12:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:12:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:12:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:12:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:12:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:12:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:12:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:12:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:12:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:12:45.859 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:12:45.859 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:12:45.859 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:12:45.859 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2023-12-02T03:12:45.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-02T03:12:45.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:12:45.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:12:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_24 2023-12-02T03:12:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T03:12:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T03:12:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T03:12:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T03:12:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T03:12:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T03:12:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:12:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T03:12:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T03:12:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T03:12:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T03:12:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:12:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T03:12:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T03:12:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T03:12:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T03:12:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:12:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:12:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:12:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_24 2023-12-02T03:12:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-02T03:12:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:12:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:12:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_24 2023-12-02T03:12:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T03:12:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T03:12:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T03:12:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T03:12:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T03:12:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T03:12:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:12:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T03:12:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T03:12:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T03:12:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T03:12:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:12:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T03:12:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T03:12:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T03:12:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T03:12:46.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:12:46.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:12:46.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:12:46.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_24 2023-12-02T03:12:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-02T03:12:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:12:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:12:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:12:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 8 2023-12-02T03:12:46.264 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~276s 2023-12-02T03:12:51.667 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~281s 2023-12-02T03:12:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:12:54.157 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:12:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2023-12-02T03:12:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:12:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:12:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:12:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:12:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:12:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:12:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:12:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:12:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:12:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:12:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:12:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:12:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:12:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:12:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:12:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:12:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:12:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2023-12-02T03:12:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-02T03:12:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:12:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:12:54.274 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:54.274 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2023-12-02T03:12:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:12:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:12:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:12:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:12:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:12:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:12:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:12:54.276 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:12:54.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:12:54.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:54.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:54.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:12:54.277 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:12:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:12:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:12:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:12:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:12:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:12:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:12:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:12:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:12:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2023-12-02T03:12:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-02T03:12:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:12:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:12:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_24 2023-12-02T03:12:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T03:12:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T03:12:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T03:12:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T03:12:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T03:12:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T03:12:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:12:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T03:12:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T03:12:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T03:12:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T03:12:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:12:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T03:12:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T03:12:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T03:12:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T03:12:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:12:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:12:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:12:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_24 2023-12-02T03:12:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-02T03:12:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:12:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:12:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:12:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_24 2023-12-02T03:12:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T03:12:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:12:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:12:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T03:12:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T03:12:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:12:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:12:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:12:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T03:12:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T03:12:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:12:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T03:12:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:12:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T03:12:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T03:12:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:12:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:12:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T03:12:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T03:12:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:12:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:12:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:12:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:12:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T03:12:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:12:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:12:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T03:12:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T03:12:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T03:12:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:12:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:12:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:12:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_24 2023-12-02T03:12:54.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-02T03:12:54.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:12:54.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:12:54.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:12:54.590 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 8 2023-12-02T03:12:57.071 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~286s 2023-12-02T03:13:02.474 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~292s 2023-12-02T03:13:02.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:13:02.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:13:02.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:13:02.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2023-12-02T03:13:02.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:13:02.595 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:13:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:13:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:13:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:13:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:13:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:13:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:13:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:13:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:13:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:13:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:13:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:13:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:13:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:13:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:13:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:13:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:13:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:13:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:13:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:13:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:13:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:13:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:13:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:13:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:13:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:13:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:13:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:13:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:13:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:13:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:13:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:13:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2023-12-02T03:13:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-02T03:13:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:13:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:13:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:13:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2023-12-02T03:13:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:13:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:13:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:13:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:13:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:13:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:13:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:13:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:13:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:13:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:13:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:13:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:13:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:13:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:13:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:13:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:13:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:13:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:13:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:13:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:13:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:13:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:13:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:13:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:13:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:13:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:13:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:13:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:13:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:13:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:13:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:13:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:13:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:13:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2023-12-02T03:13:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-02T03:13:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:13:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:13:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:13:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_24 2023-12-02T03:13:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T03:13:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:13:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:13:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T03:13:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T03:13:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:13:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:13:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:13:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T03:13:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T03:13:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:13:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:13:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T03:13:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:13:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T03:13:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T03:13:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:13:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:13:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T03:13:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T03:13:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:13:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:13:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:13:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:13:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T03:13:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:13:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:13:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T03:13:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T03:13:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T03:13:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:13:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:13:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:13:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_24 2023-12-02T03:13:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-02T03:13:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:13:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:13:02.930 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:13:02.930 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_24 2023-12-02T03:13:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T03:13:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:13:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:13:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T03:13:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T03:13:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:13:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:13:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:13:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T03:13:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T03:13:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:13:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:13:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T03:13:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:13:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T03:13:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T03:13:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:13:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:13:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T03:13:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T03:13:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:13:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:13:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:13:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:13:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T03:13:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:13:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:13:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T03:13:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T03:13:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T03:13:02.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:13:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:13:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:13:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_24 2023-12-02T03:13:03.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T03:13:03.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T03:13:03.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T03:13:03.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T03:13:03.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T03:13:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T03:13:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T03:13:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:13:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2023-12-02T03:13:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:13:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:13:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_24 2023-12-02T03:13:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T03:13:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T03:13:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2023-12-02T03:13:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:13:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:13:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_24 2023-12-02T03:13:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-image_24.status 2023-12-02T03:13:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-image_24.status 2023-12-02T03:13:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:13:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_24.status 2023-12-02T03:13:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-image_24.status 2023-12-02T03:13:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-image_24.status 2023-12-02T03:13:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:13:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_24.status 2023-12-02T03:13:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2023-12-02T03:13:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:13:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_24 2023-12-02T03:13:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2023-12-02T03:13:03.313 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 165 2023-12-02T03:13:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.kebs98RNzE/cluster2-mirror-image_24.status 2023-12-02T03:13:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:13:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_24 2023-12-02T03:13:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_24.status 2023-12-02T03:13:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2023-12-02T03:13:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:13:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T03:13:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=165 2023-12-02T03:13:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:13:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:13:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T03:13:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_24.status 2023-12-02T03:13:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:13:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_24 2023-12-02T03:13:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_24.status 2023-12-02T03:13:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-02T03:13:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:13:03.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:13:03.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:13:03.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:13:03.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:13:03.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:13:03.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:13:03.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 165 = '' 2023-12-02T03:13:03.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:13:03.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T03:13:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_24.status 2023-12-02T03:13:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:13:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_24 2023-12-02T03:13:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_24.status 2023-12-02T03:13:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-02T03:13:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:13:04.073 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:13:04.073 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:13:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:13:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:13:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:13:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:13:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 165 = '' 2023-12-02T03:13:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:13:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T03:13:04.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_24.status 2023-12-02T03:13:04.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:13:04.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:04.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_24 2023-12-02T03:13:04.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_24.status 2023-12-02T03:13:04.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-02T03:13:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:13:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:13:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=165 2023-12-02T03:13:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 165 = 165 2023-12-02T03:13:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:13:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2023-12-02T03:13:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:13:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_24 2023-12-02T03:13:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:13:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:13:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T03:13:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:13:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2023-12-02T03:13:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_24': 2023-12-02T03:13:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:13:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:13:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:13:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12c05c91ef70 2023-12-02T03:13:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12c05c91ef70 2023-12-02T03:13:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:13:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:13:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:13:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:13:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:04:02 2023 2023-12-02T03:13:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:04:02 2023 2023-12-02T03:13:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:04:02 2023 2023-12-02T03:13:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T03:13:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:13:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:13:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 9b3405db-a7dc-43f6-8462-f7325ff11d39 2023-12-02T03:13:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:13:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:13:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror image_24 snap 2023-12-02T03:13:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_24 2023-12-02T03:13:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:13:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T03:13:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_24.export 2023-12-02T03:13:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_24.export 2023-12-02T03:13:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_24.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_24.export 2023-12-02T03:13:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.kebs98RNzE/cluster2-mirror-image_24.export 2023-12-02T03:13:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:13:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.kebs98RNzE/cluster1-mirror-image_24.export 2023-12-02T03:13:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:13:06.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-image_24.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_24.export 2023-12-02T03:13:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_24.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_24.export 2023-12-02T03:13:06.484 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:13:06.484 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:13:06.484 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_25 2023-12-02T03:13:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror image_25 snap 2023-12-02T03:13:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:13:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_25 2023-12-02T03:13:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:13:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T03:13:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2023-12-02T03:13:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:13:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_25 2023-12-02T03:13:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2023-12-02T03:13:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:13:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_25 2023-12-02T03:13:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T03:13:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:13:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:13:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:13:07.878 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~297s 2023-12-02T03:13:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2023-12-02T03:13:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:13:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_25 2023-12-02T03:13:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T03:13:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T03:13:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T03:13:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2023-12-02T03:13:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T03:13:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:13:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T03:13:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T03:13:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T03:13:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T03:13:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:13:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T03:13:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:13:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:13:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:13:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2023-12-02T03:13:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:13:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:13:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:13:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:13:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:13:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:13:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:13:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:13:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:13:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:13:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:13:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:13:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:13:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:13:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:13:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:13:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:13:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:13:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:13:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:13:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:13:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:13:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:13:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:13:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:13:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:13:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:13:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:13:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:13:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:13:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:13:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:13:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:13:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2023-12-02T03:13:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2023-12-02T03:13:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:13:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:13:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:13:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2023-12-02T03:13:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:13:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:13:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:13:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:13:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:13:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:13:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:13:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:13:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:13:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:13:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:13:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:13:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:13:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:13:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:13:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:13:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:13:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:13:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:13:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:13:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:13:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:13:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:13:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:13:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:13:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:13:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:13:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:13:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:13:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:13:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:13:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:13:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:13:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2023-12-02T03:13:08.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2023-12-02T03:13:08.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:13:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:13:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:13:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_25 2023-12-02T03:13:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T03:13:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:13:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:13:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T03:13:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T03:13:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:13:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:13:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:13:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T03:13:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T03:13:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:13:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:13:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T03:13:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:13:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T03:13:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T03:13:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:13:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:13:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T03:13:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T03:13:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:13:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:13:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:13:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:13:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T03:13:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:13:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:13:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T03:13:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T03:13:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T03:13:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:13:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:13:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:13:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_25 2023-12-02T03:13:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2023-12-02T03:13:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:13:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:13:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:13:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_25 2023-12-02T03:13:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T03:13:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:13:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:13:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T03:13:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T03:13:08.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:13:08.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:13:08.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:13:08.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T03:13:08.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T03:13:08.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:13:08.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:13:08.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T03:13:08.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:13:08.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-02T03:13:08.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:3 2023-12-02T03:13:08.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:13:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:13:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-02T03:13:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:3 2023-12-02T03:13:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:13:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:13:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:13:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T03:13:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-02T03:13:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:13:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:13:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=3 2023-12-02T03:13:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=3 2023-12-02T03:13:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-02T03:13:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:13:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:13:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-02T03:13:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_25 2023-12-02T03:13:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T03:13:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T03:13:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T03:13:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T03:13:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T03:13:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T03:13:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T03:13:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:13:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2023-12-02T03:13:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:13:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:13:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_25 2023-12-02T03:13:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T03:13:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T03:13:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2023-12-02T03:13:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:13:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:13:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_25 2023-12-02T03:13:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-image_25.status 2023-12-02T03:13:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-image_25.status 2023-12-02T03:13:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:13:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_25.status 2023-12-02T03:13:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-image_25.status 2023-12-02T03:13:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-image_25.status 2023-12-02T03:13:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:13:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_25.status 2023-12-02T03:13:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2023-12-02T03:13:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:13:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_25 2023-12-02T03:13:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2023-12-02T03:13:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Snapshot ID: 167 2023-12-02T03:13:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.kebs98RNzE/cluster2-mirror-image_25.status 2023-12-02T03:13:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:13:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_25 2023-12-02T03:13:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_25.status 2023-12-02T03:13:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2023-12-02T03:13:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:13:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-02T03:13:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snapshot_id=167 2023-12-02T03:13:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:13:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:13:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.2 2023-12-02T03:13:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_25.status 2023-12-02T03:13:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:13:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_25 2023-12-02T03:13:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_25.status 2023-12-02T03:13:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-02T03:13:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:13:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:13:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:13:09.709 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:13:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:13:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:13:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:13:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 167 = '' 2023-12-02T03:13:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:13:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.4 2023-12-02T03:13:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_25.status 2023-12-02T03:13:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:13:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_25 2023-12-02T03:13:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_25.status 2023-12-02T03:13:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-02T03:13:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:13:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ true 2023-12-02T03:13:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:13:10.164 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:-:1.1: Document is empty 2023-12-02T03:13:10.164 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: 2023-12-02T03:13:10.165 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:^ 2023-12-02T03:13:10.165 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id= 2023-12-02T03:13:10.165 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 167 = '' 2023-12-02T03:13:10.165 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-02T03:13:10.165 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 0.8 2023-12-02T03:13:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.kebs98RNzE/cluster1-mirror-image_25.status 2023-12-02T03:13:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:13:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_25 2023-12-02T03:13:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_25.status 2023-12-02T03:13:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-02T03:13:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-02T03:13:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-02T03:13:11.015 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local primary_snapshot_id=167 2023-12-02T03:13:11.015 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test 167 = 167 2023-12-02T03:13:11.015 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:13:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2023-12-02T03:13:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:13:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_25 2023-12-02T03:13:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:13:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:13:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-02T03:13:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:13:12.016 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2023-12-02T03:13:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_25': 2023-12-02T03:13:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:13:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:13:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:13:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 11b8c0e8dff8 2023-12-02T03:13:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.11b8c0e8dff8 2023-12-02T03:13:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:13:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:13:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:13:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:13:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:04:12 2023 2023-12-02T03:13:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:04:12 2023 2023-12-02T03:13:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:04:12 2023 2023-12-02T03:13:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: protected: False 2023-12-02T03:13:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:13:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:13:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 26af4e3d-22a4-46d1-af96-3f3836056461 2023-12-02T03:13:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:13:12.069 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:13:12.069 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ compare_image_snaps mirror image_25 snap 2023-12-02T03:13:12.069 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:12.069 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_25 2023-12-02T03:13:12.069 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:13:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret=0 2023-12-02T03:13:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local rmt_export=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_25.export 2023-12-02T03:13:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local loc_export=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_25.export 2023-12-02T03:13:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_25.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_25.export 2023-12-02T03:13:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.kebs98RNzE/cluster2-mirror-image_25.export 2023-12-02T03:13:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:13:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.kebs98RNzE/cluster1-mirror-image_25.export 2023-12-02T03:13:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-02T03:13:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cmp /tmp/tmp.kebs98RNzE/cluster2-mirror-image_25.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_25.export 2023-12-02T03:13:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -f /tmp/tmp.kebs98RNzE/cluster2-mirror-image_25.export /tmp/tmp.kebs98RNzE/cluster1-mirror-image_25.export 2023-12-02T03:13:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:13:12.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-02T03:13:12.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ image=image_26 2023-12-02T03:13:12.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ create_snap cluster2 mirror image_26 snap 2023-12-02T03:13:12.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:13:12.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:12.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_26 2023-12-02T03:13:12.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local snap_name=snap 2023-12-02T03:13:12.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2023-12-02T03:13:13.285 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~303s 2023-12-02T03:13:13.285 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-12-02T03:13:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2023-12-02T03:13:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:13:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_26 2023-12-02T03:13:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2023-12-02T03:13:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:13:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_26 2023-12-02T03:13:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local state=started 2023-12-02T03:13:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local s 2023-12-02T03:13:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:13:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ sleep 1 2023-12-02T03:13:13.687 INFO:tasks.ceph.cluster2.osd.1:Sent signal 15 2023-12-02T03:13:13.687 INFO:tasks.ceph.cluster2.osd.2:Sent signal 15 2023-12-02T03:13:13.687 INFO:tasks.ceph.cluster2.mon.a:Sent signal 15 2023-12-02T03:13:13.688 INFO:tasks.ceph.cluster2.mgr.x:Sent signal 15 2023-12-02T03:13:13.688 INFO:tasks.ceph.cluster2.osd.1.smithi057.stderr:2023-12-02T03:13:13.686+0000 7fbdaa957640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 (PID: 85669) UID: 0 2023-12-02T03:13:13.688 INFO:tasks.ceph.cluster2.osd.1.smithi057.stderr:2023-12-02T03:13:13.686+0000 7fbdaa957640 -1 osd.1 209 *** Got signal Terminated *** 2023-12-02T03:13:13.688 INFO:tasks.ceph.cluster2.osd.1.smithi057.stderr:2023-12-02T03:13:13.686+0000 7fbdaa957640 -1 osd.1 209 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-02T03:13:13.689 INFO:tasks.ceph.cluster2.osd.2.smithi057.stderr:2023-12-02T03:13:13.686+0000 7f3483d8c640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 (PID: 85671) UID: 0 2023-12-02T03:13:13.689 INFO:tasks.ceph.cluster2.osd.2.smithi057.stderr:2023-12-02T03:13:13.686+0000 7f3483d8c640 -1 osd.2 209 *** Got signal Terminated *** 2023-12-02T03:13:13.689 INFO:tasks.ceph.cluster2.osd.2.smithi057.stderr:2023-12-02T03:13:13.686+0000 7f3483d8c640 -1 osd.2 209 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-02T03:13:13.689 INFO:tasks.ceph.cluster2.mon.a.smithi057.stderr:2023-12-02T03:13:13.686+0000 7f4da3c7a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster cluster2 -i a (PID: 85361) UID: 0 2023-12-02T03:13:13.689 INFO:tasks.ceph.cluster2.mon.a.smithi057.stderr:2023-12-02T03:13:13.686+0000 7f4da3c7a640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-02T03:13:13.690 INFO:tasks.ceph.cluster2.mgr.x.smithi029.stderr:2023-12-02T03:13:13.688+0000 7f1aaf815640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster cluster2 -i x (PID: 89483) UID: 0 2023-12-02T03:13:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2023-12-02T03:13:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T03:13:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_26 2023-12-02T03:13:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local test_state=started 2023-12-02T03:13:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_result 2023-12-02T03:13:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local current_state=stopped 2023-12-02T03:13:14.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2023-12-02T03:13:14.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_instance=cluster1 2023-12-02T03:13:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:13:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ grep -i state 2023-12-02T03:13:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1 2023-12-02T03:13:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance=cluster1 2023-12-02T03:13:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local loop_instance 2023-12-02T03:13:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-02T03:13:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sleep 0 2023-12-02T03:13:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-02T03:13:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ seq 0 3 2023-12-02T03:13:14.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:13:14.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2023-12-02T03:13:14.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T03:13:14.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:13:14.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:13:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T03:13:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T03:13:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:13:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:13:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:13:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T03:13:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T03:13:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:13:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:13:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T03:13:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:13:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-02T03:13:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:0 2023-12-02T03:13:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:13:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:13:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-02T03:13:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:0 2023-12-02T03:13:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:13:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:13:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:13:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T03:13:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-02T03:13:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:13:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:13:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=0 2023-12-02T03:13:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=0 2023-12-02T03:13:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-02T03:13:14.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:13:14.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:13:14.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-02T03:13:14.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2023-12-02T03:13:14.548 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2023-12-02T03:13:14.548 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:13:14.548 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:13:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:13:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2023-12-02T03:13:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T03:13:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:13:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:13:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T03:13:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T03:13:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:13:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:13:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:13:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T03:13:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T03:13:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:13:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:13:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T03:13:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:13:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-02T03:13:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:1 2023-12-02T03:13:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:13:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:13:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-02T03:13:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:1 2023-12-02T03:13:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:13:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:13:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:13:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T03:13:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-02T03:13:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:13:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:13:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=1 2023-12-02T03:13:14.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=1 2023-12-02T03:13:14.554 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-02T03:13:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:13:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:13:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-02T03:13:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2023-12-02T03:13:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2023-12-02T03:13:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd mirror status : get status for rbd mirror 2023-12-02T03:13:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:admin_socket: invalid command 2023-12-02T03:13:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T03:13:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_26 2023-12-02T03:13:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T03:13:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ shift 2023-12-02T03:13:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T03:13:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T03:13:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T03:13:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T03:13:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T03:13:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T03:13:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T03:13:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T03:13:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T03:13:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:13:14.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T03:13:14.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:13:14.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-02T03:13:14.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local local_cluster=cluster1:2 2023-12-02T03:13:14.663 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster=cluster1 2023-12-02T03:13:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance 2023-12-02T03:13:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-02T03:13:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local val=cluster1:2 2023-12-02T03:13:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local cluster_var_name=local_cluster 2023-12-02T03:13:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ local instance_var_name=instance 2023-12-02T03:13:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T03:13:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T03:13:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-02T03:13:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval local_cluster=cluster1 2023-12-02T03:13:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ local_cluster=cluster1 2023-12-02T03:13:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ eval instance=2 2023-12-02T03:13:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ instance=2 2023-12-02T03:13:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-02T03:13:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:13:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:13:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-02T03:13:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_26 2023-12-02T03:13:14.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ return 0 2023-12-02T03:13:14.772 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status_result=' "state": "Replaying"' 2023-12-02T03:13:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo ' "state": "Replaying"' 2023-12-02T03:13:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ grep -i Replaying 2023-12-02T03:13:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "state": "Replaying" 2023-12-02T03:13:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ current_state=started 2023-12-02T03:13:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test started = started 2023-12-02T03:13:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 0 2023-12-02T03:13:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2023-12-02T03:13:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:13:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:13:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_26 2023-12-02T03:13:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = journal ']' 2023-12-02T03:13:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' snapshot = snapshot ']' 2023-12-02T03:13:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2023-12-02T03:13:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_cluster=cluster1 2023-12-02T03:13:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:13:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_26 2023-12-02T03:13:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster2-mirror-image_26.status 2023-12-02T03:13:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster2-mirror-image_26.status 2023-12-02T03:13:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:13:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local status_log=/tmp/tmp.kebs98RNzE/cluster2-mirror-image_26.status 2023-12-02T03:13:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ mkfname cluster1-mirror-image_26.status 2023-12-02T03:13:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo cluster1-mirror-image_26.status 2023-12-02T03:13:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-02T03:13:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local local_status_log=/tmp/tmp.kebs98RNzE/cluster1-mirror-image_26.status 2023-12-02T03:13:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2023-12-02T03:13:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T03:13:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pool=mirror 2023-12-02T03:13:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local image=image_26 2023-12-02T03:13:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2023-12-02T03:13:18.165 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9343301229500015) > weight (0.25) 2023-12-02T03:13:18.165 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-02T03:13:18.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-02T03:13:18.165 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-12-02T03:13:18.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi057.stderr:2023-12-02T03:13:18.164+0000 7fd23b295640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 106520) UID: 1000 2023-12-02T03:13:20.166 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-02T03:13:20.166 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T03:13:20.167 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-02T03:13:20.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-02T03:13:20.167 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-02T03:13:20.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-02T03:13:20.170 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2023-12-02T03:13:41.053 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5400610678388006) > weight (0.25) 2023-12-02T03:13:41.053 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8049622165095303) > weight (0.25) 2023-12-02T03:13:41.053 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-02T03:13:41.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-02T03:13:41.054 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-02T03:13:41.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr:2023-12-02T03:13:41.053+0000 7f385b071640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 96585) UID: 1000 2023-12-02T03:13:45.054 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-02T03:13:45.055 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T03:15:18.171 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-02T03:15:18.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-02T03:15:18.172 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-02T03:15:18.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr:2023-12-02T03:15:18.172+0000 7f30e88ca640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 103486) UID: 1000 2023-12-02T03:15:19.173 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-02T03:15:19.173 INFO:teuthology.orchestra.run:waiting for 600 2023-12-02T03:18:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: couldn't connect to the cluster! 2023-12-02T03:18:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ cleanup 110 2023-12-02T03:18:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local error_code=110 2023-12-02T03:18:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ set +e 2023-12-02T03:18:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 110 -ne 0 ']' 2023-12-02T03:18:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ status 2023-12-02T03:18:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster daemon image_pool image_ns image 2023-12-02T03:18:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for cluster in ${CLUSTER1} ${CLUSTER2} 2023-12-02T03:18:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster1 status 2023-12-02T03:18:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster1 status' 2023-12-02T03:18:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T03:18:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ ceph --cluster cluster1 -s 2023-12-02T03:18:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: cluster: 2023-12-02T03:18:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 69e9688d-4570-4fa3-a52d-d6c0eaea7d57 2023-12-02T03:18:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: health: HEALTH_OK 2023-12-02T03:18:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: services: 2023-12-02T03:18:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mon: 1 daemons, quorum a (age 30m) 2023-12-02T03:18:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mgr: x(active, since 30m) 2023-12-02T03:18:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: osd: 3 osds: 3 up (since 30m), 3 in (since 30m) 2023-12-02T03:18:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: rbd-mirror: 4 daemons active (1 hosts) 2023-12-02T03:18:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: data: 2023-12-02T03:18:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: pools: 4 pools, 137 pgs 2023-12-02T03:18:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: objects: 983 objects, 3.1 GiB 2023-12-02T03:18:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: usage: 6.4 GiB used, 264 GiB / 270 GiB avail 2023-12-02T03:18:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: pgs: 137 active+clean 2023-12-02T03:18:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: io: 2023-12-02T03:18:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: client: 2.6 KiB/s rd, 1023 B/s wr, 2 op/s rd, 0 op/s wr 2023-12-02T03:18:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:15.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T03:18:15.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ ceph --cluster cluster1 service dump 2023-12-02T03:18:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:{ 2023-12-02T03:18:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "epoch": 26, 2023-12-02T03:18:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "modified": "2023-12-02T03:15:22.241493+0000", 2023-12-02T03:18:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "services": { 2023-12-02T03:18:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "rbd-mirror": { 2023-12-02T03:18:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "daemons": { 2023-12-02T03:18:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "summary": "", 2023-12-02T03:18:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "4533": { 2023-12-02T03:18:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "start_epoch": 11, 2023-12-02T03:18:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "start_stamp": "2023-12-02T02:55:17.025927+0000", 2023-12-02T03:18:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "gid": 4533, 2023-12-02T03:18:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "addr": "172.21.15.57:0/899731226", 2023-12-02T03:18:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "metadata": { 2023-12-02T03:18:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "arch": "x86_64", 2023-12-02T03:18:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "ceph_release": "reef", 2023-12-02T03:18:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "ceph_version": "ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev)", 2023-12-02T03:18:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "ceph_version_short": "18.0.0-7631-gbed76814", 2023-12-02T03:18:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2023-12-02T03:18:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "distro": "centos", 2023-12-02T03:18:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "distro_description": "CentOS Stream 9", 2023-12-02T03:18:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "distro_version": "9", 2023-12-02T03:18:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "hostname": "smithi057", 2023-12-02T03:18:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "id": "mirror.2", 2023-12-02T03:18:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "instance_id": "4533", 2023-12-02T03:18:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Tue Nov 21 10:17:52 UTC 2023", 2023-12-02T03:18:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "kernel_version": "5.14.0-388.el9.x86_64", 2023-12-02T03:18:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "mem_swap_kb": "0", 2023-12-02T03:18:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "mem_total_kb": "32494812", 2023-12-02T03:18:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "os": "Linux" 2023-12-02T03:18:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: }, 2023-12-02T03:18:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "task_status": {} 2023-12-02T03:18:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: }, 2023-12-02T03:18:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "4782": { 2023-12-02T03:18:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "start_epoch": 17, 2023-12-02T03:18:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "start_stamp": "2023-12-02T03:02:27.159868+0000", 2023-12-02T03:18:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "gid": 4782, 2023-12-02T03:18:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "addr": "172.21.15.57:0/170685927", 2023-12-02T03:18:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "metadata": { 2023-12-02T03:18:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "arch": "x86_64", 2023-12-02T03:18:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "ceph_release": "reef", 2023-12-02T03:18:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "ceph_version": "ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev)", 2023-12-02T03:18:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "ceph_version_short": "18.0.0-7631-gbed76814", 2023-12-02T03:18:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2023-12-02T03:18:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "distro": "centos", 2023-12-02T03:18:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "distro_description": "CentOS Stream 9", 2023-12-02T03:18:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "distro_version": "9", 2023-12-02T03:18:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "hostname": "smithi057", 2023-12-02T03:18:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "id": "mirror.1", 2023-12-02T03:18:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "instance_id": "4782", 2023-12-02T03:18:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Tue Nov 21 10:17:52 UTC 2023", 2023-12-02T03:18:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "kernel_version": "5.14.0-388.el9.x86_64", 2023-12-02T03:18:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "mem_swap_kb": "0", 2023-12-02T03:18:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "mem_total_kb": "32494812", 2023-12-02T03:18:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "os": "Linux" 2023-12-02T03:18:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: }, 2023-12-02T03:18:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "task_status": {} 2023-12-02T03:18:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: }, 2023-12-02T03:18:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "4813": { 2023-12-02T03:18:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "start_epoch": 21, 2023-12-02T03:18:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "start_stamp": "2023-12-02T03:07:21.167331+0000", 2023-12-02T03:18:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "gid": 4813, 2023-12-02T03:18:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "addr": "172.21.15.57:0/2024383480", 2023-12-02T03:18:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "metadata": { 2023-12-02T03:18:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "arch": "x86_64", 2023-12-02T03:18:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "ceph_release": "reef", 2023-12-02T03:18:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "ceph_version": "ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev)", 2023-12-02T03:18:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "ceph_version_short": "18.0.0-7631-gbed76814", 2023-12-02T03:18:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2023-12-02T03:18:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "distro": "centos", 2023-12-02T03:18:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "distro_description": "CentOS Stream 9", 2023-12-02T03:18:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "distro_version": "9", 2023-12-02T03:18:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "hostname": "smithi057", 2023-12-02T03:18:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "id": "mirror.3", 2023-12-02T03:18:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "instance_id": "4813", 2023-12-02T03:18:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Tue Nov 21 10:17:52 UTC 2023", 2023-12-02T03:18:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "kernel_version": "5.14.0-388.el9.x86_64", 2023-12-02T03:18:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "mem_swap_kb": "0", 2023-12-02T03:18:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "mem_total_kb": "32494812", 2023-12-02T03:18:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "os": "Linux" 2023-12-02T03:18:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: }, 2023-12-02T03:18:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "task_status": {} 2023-12-02T03:18:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: }, 2023-12-02T03:18:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "5226": { 2023-12-02T03:18:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "start_epoch": 25, 2023-12-02T03:18:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "start_stamp": "2023-12-02T03:12:37.145347+0000", 2023-12-02T03:18:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "gid": 5226, 2023-12-02T03:18:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "addr": "172.21.15.57:0/497330572", 2023-12-02T03:18:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "metadata": { 2023-12-02T03:18:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "arch": "x86_64", 2023-12-02T03:18:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "ceph_release": "reef", 2023-12-02T03:18:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "ceph_version": "ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev)", 2023-12-02T03:18:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "ceph_version_short": "18.0.0-7631-gbed76814", 2023-12-02T03:18:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2023-12-02T03:18:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "distro": "centos", 2023-12-02T03:18:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "distro_description": "CentOS Stream 9", 2023-12-02T03:18:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "distro_version": "9", 2023-12-02T03:18:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "hostname": "smithi057", 2023-12-02T03:18:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "id": "mirror.0", 2023-12-02T03:18:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "instance_id": "5226", 2023-12-02T03:18:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Tue Nov 21 10:17:52 UTC 2023", 2023-12-02T03:18:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "kernel_version": "5.14.0-388.el9.x86_64", 2023-12-02T03:18:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "mem_swap_kb": "0", 2023-12-02T03:18:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "mem_total_kb": "32494812", 2023-12-02T03:18:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "os": "Linux" 2023-12-02T03:18:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: }, 2023-12-02T03:18:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "task_status": {} 2023-12-02T03:18:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: } 2023-12-02T03:18:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: } 2023-12-02T03:18:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: } 2023-12-02T03:18:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: } 2023-12-02T03:18:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:} 2023-12-02T03:18:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T03:18:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ ceph --cluster cluster1 service status 2023-12-02T03:18:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:{ 2023-12-02T03:18:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "rbd-mirror": { 2023-12-02T03:18:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "4533": { 2023-12-02T03:18:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "status_stamp": "2023-12-02T03:17:52.090838+0000", 2023-12-02T03:18:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "last_beacon": "2023-12-02T03:18:12.091739+0000", 2023-12-02T03:18:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "status": { 2023-12-02T03:18:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":13,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4536\"},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4539\"}}" 2023-12-02T03:18:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: } 2023-12-02T03:18:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: }, 2023-12-02T03:18:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "4782": { 2023-12-02T03:18:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "status_stamp": "2023-12-02T03:18:02.201642+0000", 2023-12-02T03:18:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "last_beacon": "2023-12-02T03:18:12.202056+0000", 2023-12-02T03:18:16.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "status": { 2023-12-02T03:18:16.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":13,\"image_error_count\":0,\"image_local_count\":50,\"image_remote_count\":50,\"image_warning_count\":0,\"instance_id\":\"4785\",\"leader\":true,\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_local_count\":0,\"image_remote_count\":0,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_local_count\":0,\"image_remote_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_local_count\":0,\"image_remote_count\":0,\"image_warning_count\":0,\"instance_id\":\"4788\",\"leader\":true}}" 2023-12-02T03:18:16.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: } 2023-12-02T03:18:16.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: }, 2023-12-02T03:18:16.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "4813": { 2023-12-02T03:18:16.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "status_stamp": "2023-12-02T03:17:56.196149+0000", 2023-12-02T03:18:16.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "last_beacon": "2023-12-02T03:18:11.196844+0000", 2023-12-02T03:18:16.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "status": { 2023-12-02T03:18:16.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":12,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4816\",\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4818\"}}" 2023-12-02T03:18:16.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: } 2023-12-02T03:18:16.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: }, 2023-12-02T03:18:16.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "5226": { 2023-12-02T03:18:16.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "status_stamp": "2023-12-02T03:18:12.161508+0000", 2023-12-02T03:18:16.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "last_beacon": "2023-12-02T03:18:12.161508+0000", 2023-12-02T03:18:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "status": { 2023-12-02T03:18:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":12,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"5229\",\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"5232\"}}" 2023-12-02T03:18:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: } 2023-12-02T03:18:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: } 2023-12-02T03:18:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: } 2023-12-02T03:18:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:} 2023-12-02T03:18:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image_pool in ${POOL} ${PARENT_POOL} 2023-12-02T03:18:16.099 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-12-02T03:18:16.099 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster1 mirror images 2023-12-02T03:18:16.099 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster1 mirror images' 2023-12-02T03:18:16.099 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' ls -l 2023-12-02T03:18:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:NAME SIZE PARENT FMT PROT LOCK 2023-12-02T03:18:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_1 128 MiB 2 2023-12-02T03:18:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_1@snap 128 MiB 2 2023-12-02T03:18:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_10 128 MiB 2 2023-12-02T03:18:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_10@snap 128 MiB 2 2023-12-02T03:18:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_11 128 MiB 2 2023-12-02T03:18:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_11@snap 128 MiB 2 2023-12-02T03:18:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12 128 MiB 2 2023-12-02T03:18:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12@snap 128 MiB 2 2023-12-02T03:18:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_13 128 MiB 2 2023-12-02T03:18:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_13@snap 128 MiB 2 2023-12-02T03:18:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_14 128 MiB 2 2023-12-02T03:18:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_14@snap 128 MiB 2 2023-12-02T03:18:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_15 128 MiB 2 2023-12-02T03:18:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_15@snap 128 MiB 2 2023-12-02T03:18:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_16 128 MiB 2 2023-12-02T03:18:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_16@snap 128 MiB 2 2023-12-02T03:18:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_17 128 MiB 2 2023-12-02T03:18:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_17@snap 128 MiB 2 2023-12-02T03:18:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_18 128 MiB 2 2023-12-02T03:18:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_18@snap 128 MiB 2 2023-12-02T03:18:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_19 128 MiB 2 2023-12-02T03:18:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_19@snap 128 MiB 2 2023-12-02T03:18:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_2 128 MiB 2 2023-12-02T03:18:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_2@snap 128 MiB 2 2023-12-02T03:18:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_20 128 MiB 2 2023-12-02T03:18:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_20@snap 128 MiB 2 2023-12-02T03:18:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_21 128 MiB 2 2023-12-02T03:18:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_21@snap 128 MiB 2 2023-12-02T03:18:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_22 128 MiB 2 2023-12-02T03:18:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_22@snap 128 MiB 2 2023-12-02T03:18:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_23 128 MiB 2 2023-12-02T03:18:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_23@snap 128 MiB 2 2023-12-02T03:18:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_24 128 MiB 2 2023-12-02T03:18:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_24@snap 128 MiB 2 2023-12-02T03:18:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_25 128 MiB 2 2023-12-02T03:18:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_25@snap 128 MiB 2 2023-12-02T03:18:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_26 128 MiB 2 2023-12-02T03:18:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_27 128 MiB 2 2023-12-02T03:18:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_28 128 MiB 2 2023-12-02T03:18:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_29 128 MiB 2 2023-12-02T03:18:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_3 128 MiB 2 2023-12-02T03:18:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_3@snap 128 MiB 2 2023-12-02T03:18:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_30 128 MiB 2 2023-12-02T03:18:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_31 128 MiB 2 2023-12-02T03:18:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_32 128 MiB 2 2023-12-02T03:18:16.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_33 128 MiB 2 2023-12-02T03:18:16.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_34 128 MiB 2 2023-12-02T03:18:16.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_35 128 MiB 2 2023-12-02T03:18:16.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_36 128 MiB 2 2023-12-02T03:18:16.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_37 128 MiB 2 2023-12-02T03:18:16.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_38 128 MiB 2 2023-12-02T03:18:16.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_39 128 MiB 2 2023-12-02T03:18:16.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_4 128 MiB 2 2023-12-02T03:18:16.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_4@snap 128 MiB 2 2023-12-02T03:18:16.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_40 128 MiB 2 2023-12-02T03:18:16.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_41 128 MiB 2 2023-12-02T03:18:16.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_42 128 MiB 2 2023-12-02T03:18:16.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_43 128 MiB 2 2023-12-02T03:18:16.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_44 128 MiB 2 2023-12-02T03:18:16.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_45 128 MiB 2 2023-12-02T03:18:16.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_46 128 MiB 2 2023-12-02T03:18:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_47 128 MiB 2 2023-12-02T03:18:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_48 128 MiB 2 2023-12-02T03:18:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_49 128 MiB 2 2023-12-02T03:18:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_5 128 MiB 2 2023-12-02T03:18:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_5@snap 128 MiB 2 2023-12-02T03:18:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_50 128 MiB 2 2023-12-02T03:18:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_6 128 MiB 2 2023-12-02T03:18:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_6@snap 128 MiB 2 2023-12-02T03:18:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_7 128 MiB 2 2023-12-02T03:18:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_7@snap 128 MiB 2 2023-12-02T03:18:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_8 128 MiB 2 2023-12-02T03:18:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_8@snap 128 MiB 2 2023-12-02T03:18:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_9 128 MiB 2 2023-12-02T03:18:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_9@snap 128 MiB 2 2023-12-02T03:18:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster1 mirror mirror pool info 2023-12-02T03:18:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster1 mirror mirror pool info' 2023-12-02T03:18:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' mirror pool info 2023-12-02T03:18:16.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mode: image 2023-12-02T03:18:16.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Site Name: cluster1 2023-12-02T03:18:16.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Peer Sites: 2023-12-02T03:18:16.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:UUID: 06fbe0df-d787-44c2-aede-d6f6ace95f83 2023-12-02T03:18:16.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Name: cluster2 2023-12-02T03:18:16.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Mirror UUID: 8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:16.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Direction: rx-tx 2023-12-02T03:18:16.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:Client: client.mirror 2023-12-02T03:18:16.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:16.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster1 mirror mirror pool status' 2023-12-02T03:18:16.304 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster1 mirror mirror pool status 2023-12-02T03:18:16.304 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T03:18:16.304 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' mirror pool status --verbose 2023-12-02T03:18:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:health: WARNING 2023-12-02T03:18:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:daemon health: OK 2023-12-02T03:18:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image health: WARNING 2023-12-02T03:18:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:images: 50 total 2023-12-02T03:18:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 12 unknown 2023-12-02T03:18:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 38 starting_replay 2023-12-02T03:18:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:DAEMONS 2023-12-02T03:18:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:service 4533: 2023-12-02T03:18:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: instance_id: 4536 2023-12-02T03:18:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: client_id: mirror.2 2023-12-02T03:18:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: hostname: smithi057 2023-12-02T03:18:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: version: 18.0.0-7631-gbed76814 2023-12-02T03:18:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: leader: false 2023-12-02T03:18:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: health: OK 2023-12-02T03:18:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:service 4782: 2023-12-02T03:18:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: instance_id: 4785 2023-12-02T03:18:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: client_id: mirror.1 2023-12-02T03:18:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: hostname: smithi057 2023-12-02T03:18:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: version: 18.0.0-7631-gbed76814 2023-12-02T03:18:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: leader: true 2023-12-02T03:18:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: health: OK 2023-12-02T03:18:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:service 4813: 2023-12-02T03:18:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: instance_id: 4816 2023-12-02T03:18:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: client_id: mirror.3 2023-12-02T03:18:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: hostname: smithi057 2023-12-02T03:18:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: version: 18.0.0-7631-gbed76814 2023-12-02T03:18:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: leader: false 2023-12-02T03:18:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: health: OK 2023-12-02T03:18:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:service 5226: 2023-12-02T03:18:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: instance_id: 5229 2023-12-02T03:18:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: client_id: mirror.0 2023-12-02T03:18:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: hostname: smithi057 2023-12-02T03:18:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: version: 18.0.0-7631-gbed76814 2023-12-02T03:18:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: leader: false 2023-12-02T03:18:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: health: OK 2023-12-02T03:18:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:IMAGES 2023-12-02T03:18:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_1: 2023-12-02T03:18:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 49c16484-3472-4671-8be8-03c1846564a3 2023-12-02T03:18:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486573,"remote_snapshot_timestamp":1701486573,"replay_state":"idle"} 2023-12-02T03:18:16.360 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.0 on smithi057 2023-12-02T03:18:16.360 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:18:07 2023-12-02T03:18:16.360 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.360 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.360 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.360 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.360 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:18:16 2023-12-02T03:18:16.361 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.361 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_10: 2023-12-02T03:18:16.361 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: ddeae318-0d59-4bf2-b55e-a3f9934ab1b6 2023-12-02T03:18:16.361 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.361 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":134017024.0,"last_snapshot_bytes":134017024,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486662,"remote_snapshot_timestamp":1701486662,"replay_state":"idle"} 2023-12-02T03:18:16.361 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.3 on smithi057 2023-12-02T03:18:16.361 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:51 2023-12-02T03:18:16.361 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.361 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:18:16 2023-12-02T03:18:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_11: 2023-12-02T03:18:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 258c1b03-48a5-42f2-8745-ac0283c8bc87 2023-12-02T03:18:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":134059008.0,"last_snapshot_bytes":134059008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486670,"remote_snapshot_timestamp":1701486670,"replay_state":"idle"} 2023-12-02T03:18:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.2 on smithi057 2023-12-02T03:18:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:47 2023-12-02T03:18:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.364 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:18:16 2023-12-02T03:18:16.367 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.367 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12: 2023-12-02T03:18:16.367 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: a30b9861-e056-4151-bcfe-e4a4f2e867dd 2023-12-02T03:18:16.367 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.367 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":134189056.0,"last_snapshot_bytes":134189056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486678,"remote_snapshot_timestamp":1701486678,"replay_state":"idle"} 2023-12-02T03:18:16.367 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.3 on smithi057 2023-12-02T03:18:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:51 2023-12-02T03:18:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: down+stopped 2023-12-02T03:18:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: local image is primary 2023-12-02T03:18:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:12:50 2023-12-02T03:18:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_13: 2023-12-02T03:18:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: cdf57a21-3ee2-466a-9e94-c4dd4500ddd1 2023-12-02T03:18:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":134179840.0,"last_snapshot_bytes":134179840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486683,"remote_snapshot_timestamp":1701486683,"replay_state":"idle"} 2023-12-02T03:18:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.1 on smithi057 2023-12-02T03:18:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:57 2023-12-02T03:18:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: down+stopped 2023-12-02T03:18:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: local image is primary 2023-12-02T03:18:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:12:50 2023-12-02T03:18:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_14: 2023-12-02T03:18:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 1d600ef5-05fa-4ec8-bb36-7d891860097c 2023-12-02T03:18:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486691,"remote_snapshot_timestamp":1701486691,"replay_state":"idle"} 2023-12-02T03:18:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.0 on smithi057 2023-12-02T03:18:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:18:07 2023-12-02T03:18:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.376 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.376 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:18:16 2023-12-02T03:18:16.377 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.377 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_15: 2023-12-02T03:18:16.377 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 5a34854e-65bf-43d7-9d2c-e0481cc5a159 2023-12-02T03:18:16.377 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.377 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486699,"remote_snapshot_timestamp":1701486699,"replay_state":"idle"} 2023-12-02T03:18:16.377 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.2 on smithi057 2023-12-02T03:18:16.377 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:47 2023-12-02T03:18:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:18:16 2023-12-02T03:18:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_16: 2023-12-02T03:18:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 807f0ce9-67ef-4e3a-987e-bfe34f97a832 2023-12-02T03:18:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486705,"remote_snapshot_timestamp":1701486705,"replay_state":"idle"} 2023-12-02T03:18:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.0 on smithi057 2023-12-02T03:18:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:18:07 2023-12-02T03:18:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:48 2023-12-02T03:18:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_17: 2023-12-02T03:18:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 1f4ea1ff-7710-42b2-9726-194a70ed8f1e 2023-12-02T03:18:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486713,"remote_snapshot_timestamp":1701486713,"replay_state":"idle"} 2023-12-02T03:18:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.1 on smithi057 2023-12-02T03:18:16.380 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:57 2023-12-02T03:18:16.380 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.380 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.380 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: down+stopped 2023-12-02T03:18:16.380 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: local image is primary 2023-12-02T03:18:16.380 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:12:50 2023-12-02T03:18:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_18: 2023-12-02T03:18:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: b8d1a79d-fa19-4b64-ba6b-5a4658335ba3 2023-12-02T03:18:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":134132736.0,"last_snapshot_bytes":134132736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486719,"remote_snapshot_timestamp":1701486719,"replay_state":"idle"} 2023-12-02T03:18:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.3 on smithi057 2023-12-02T03:18:16.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:51 2023-12-02T03:18:16.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.384 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:48 2023-12-02T03:18:16.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_19: 2023-12-02T03:18:16.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: faedaa7c-e337-491d-80ec-3cfa45a9493c 2023-12-02T03:18:16.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133656576.0,"last_snapshot_bytes":133656576,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486725,"remote_snapshot_timestamp":1701486725,"replay_state":"idle"} 2023-12-02T03:18:16.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.1 on smithi057 2023-12-02T03:18:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:57 2023-12-02T03:18:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:56 2023-12-02T03:18:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_2: 2023-12-02T03:18:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 09e06fe9-a175-4009-952b-403e63e1ac10 2023-12-02T03:18:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486581,"remote_snapshot_timestamp":1701486581,"replay_state":"idle"} 2023-12-02T03:18:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.0 on smithi057 2023-12-02T03:18:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:18:07 2023-12-02T03:18:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: down+stopped 2023-12-02T03:18:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: local image is primary 2023-12-02T03:18:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:12:50 2023-12-02T03:18:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_20: 2023-12-02T03:18:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 8117a00a-84c1-4b06-96bd-60840c20cce4 2023-12-02T03:18:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486732,"remote_snapshot_timestamp":1701486732,"replay_state":"idle"} 2023-12-02T03:18:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.0 on smithi057 2023-12-02T03:18:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:18:07 2023-12-02T03:18:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:48 2023-12-02T03:18:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_21: 2023-12-02T03:18:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: b629353c-89f1-46c0-b5ff-f652e93d5cb3 2023-12-02T03:18:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486740,"remote_snapshot_timestamp":1701486740,"replay_state":"idle"} 2023-12-02T03:18:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.0 on smithi057 2023-12-02T03:18:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:18:07 2023-12-02T03:18:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:56 2023-12-02T03:18:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_22: 2023-12-02T03:18:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 8ef4933a-3981-486a-8a90-10311d67bf00 2023-12-02T03:18:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":134092800.0,"last_snapshot_bytes":134092800,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486746,"remote_snapshot_timestamp":1701486746,"replay_state":"idle"} 2023-12-02T03:18:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.1 on smithi057 2023-12-02T03:18:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:57 2023-12-02T03:18:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:48 2023-12-02T03:18:16.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_23: 2023-12-02T03:18:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 8696705f-855b-402d-afd6-1c4d55e06c91 2023-12-02T03:18:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133248000.0,"last_snapshot_bytes":133248000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486752,"remote_snapshot_timestamp":1701486752,"replay_state":"idle"} 2023-12-02T03:18:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.2 on smithi057 2023-12-02T03:18:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:47 2023-12-02T03:18:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: down+stopped 2023-12-02T03:18:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: local image is primary 2023-12-02T03:18:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:12:50 2023-12-02T03:18:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_24: 2023-12-02T03:18:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 9b3405db-a7dc-43f6-8462-f7325ff11d39 2023-12-02T03:18:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133830656.0,"last_snapshot_bytes":133830656,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486783,"remote_snapshot_timestamp":1701486783,"replay_state":"idle"} 2023-12-02T03:18:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.3 on smithi057 2023-12-02T03:18:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:51 2023-12-02T03:18:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:18:16 2023-12-02T03:18:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_25: 2023-12-02T03:18:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 26af4e3d-22a4-46d1-af96-3f3836056461 2023-12-02T03:18:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":134032384.0,"last_snapshot_bytes":134032384,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486789,"remote_snapshot_timestamp":1701486789,"replay_state":"idle"} 2023-12-02T03:18:16.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.3 on smithi057 2023-12-02T03:18:16.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:51 2023-12-02T03:18:16.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: down+stopped 2023-12-02T03:18:16.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: local image is primary 2023-12-02T03:18:16.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:12:50 2023-12-02T03:18:16.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_26: 2023-12-02T03:18:16.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 7dc110f7-a667-4d4c-9e52-4c6fe497aadb 2023-12-02T03:18:16.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486261,"remote_snapshot_timestamp":1701486261,"replay_state":"idle"} 2023-12-02T03:18:16.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.2 on smithi057 2023-12-02T03:18:16.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:47 2023-12-02T03:18:16.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.398 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:56 2023-12-02T03:18:16.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_27: 2023-12-02T03:18:16.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 453a5e01-39a4-415e-bf21-d59460038eff 2023-12-02T03:18:16.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486270,"remote_snapshot_timestamp":1701486270,"replay_state":"idle"} 2023-12-02T03:18:16.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.0 on smithi057 2023-12-02T03:18:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:18:07 2023-12-02T03:18:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:48 2023-12-02T03:18:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_28: 2023-12-02T03:18:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: e7466612-311c-416f-a7f9-bfa20e74a575 2023-12-02T03:18:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486283,"remote_snapshot_timestamp":1701486283,"replay_state":"idle"} 2023-12-02T03:18:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.1 on smithi057 2023-12-02T03:18:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:57 2023-12-02T03:18:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:18:16 2023-12-02T03:18:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_29: 2023-12-02T03:18:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 3d00c370-35ca-4837-a707-0aa3e5b4c7f4 2023-12-02T03:18:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486294,"remote_snapshot_timestamp":1701486294,"replay_state":"idle"} 2023-12-02T03:18:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.0 on smithi057 2023-12-02T03:18:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:18:07 2023-12-02T03:18:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:48 2023-12-02T03:18:16.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_3: 2023-12-02T03:18:16.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: a35ab54d-17a7-446d-984d-02da30173d31 2023-12-02T03:18:16.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133739520.0,"last_snapshot_bytes":133739520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486587,"remote_snapshot_timestamp":1701486587,"replay_state":"idle"} 2023-12-02T03:18:16.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.3 on smithi057 2023-12-02T03:18:16.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:51 2023-12-02T03:18:16.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:18:16 2023-12-02T03:18:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_30: 2023-12-02T03:18:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 03c2b3e5-9969-4a06-b152-32b7163be7b1 2023-12-02T03:18:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486307,"remote_snapshot_timestamp":1701486307,"replay_state":"idle"} 2023-12-02T03:18:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.0 on smithi057 2023-12-02T03:18:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:18:07 2023-12-02T03:18:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:18:16 2023-12-02T03:18:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_31: 2023-12-02T03:18:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: ffc4290c-3b4c-45f6-a30f-c93f3cb26275 2023-12-02T03:18:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486313,"remote_snapshot_timestamp":1701486313,"replay_state":"idle"} 2023-12-02T03:18:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.1 on smithi057 2023-12-02T03:18:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:57 2023-12-02T03:18:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:56 2023-12-02T03:18:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_32: 2023-12-02T03:18:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: d4ceac3d-c782-416e-a7e1-be445a43ae35 2023-12-02T03:18:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486328,"remote_snapshot_timestamp":1701486328,"replay_state":"idle"} 2023-12-02T03:18:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.2 on smithi057 2023-12-02T03:18:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:47 2023-12-02T03:18:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:48 2023-12-02T03:18:16.412 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.412 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_33: 2023-12-02T03:18:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 56d75e57-f434-4172-81f2-851edf5c5eb5 2023-12-02T03:18:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486339,"remote_snapshot_timestamp":1701486339,"replay_state":"idle"} 2023-12-02T03:18:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.0 on smithi057 2023-12-02T03:18:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:18:07 2023-12-02T03:18:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: down+stopped 2023-12-02T03:18:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: local image is primary 2023-12-02T03:18:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:12:50 2023-12-02T03:18:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_34: 2023-12-02T03:18:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: e2407e91-d52b-4a70-b3dd-c23362ba73c5 2023-12-02T03:18:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486353,"remote_snapshot_timestamp":1701486353,"replay_state":"idle"} 2023-12-02T03:18:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.2 on smithi057 2023-12-02T03:18:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:47 2023-12-02T03:18:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:56 2023-12-02T03:18:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_35: 2023-12-02T03:18:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 9802897e-7ccb-4bb2-8ab0-cedc09b0044e 2023-12-02T03:18:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486366,"remote_snapshot_timestamp":1701486366,"replay_state":"idle"} 2023-12-02T03:18:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.1 on smithi057 2023-12-02T03:18:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:57 2023-12-02T03:18:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:18:16 2023-12-02T03:18:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_36: 2023-12-02T03:18:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 5c374561-6563-424e-bb48-6c5014bde1fc 2023-12-02T03:18:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486377,"remote_snapshot_timestamp":1701486377,"replay_state":"idle"} 2023-12-02T03:18:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.3 on smithi057 2023-12-02T03:18:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:51 2023-12-02T03:18:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:48 2023-12-02T03:18:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_37: 2023-12-02T03:18:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 9333a204-121f-424e-849b-e6aec78f4224 2023-12-02T03:18:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486391,"remote_snapshot_timestamp":1701486391,"replay_state":"idle"} 2023-12-02T03:18:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.2 on smithi057 2023-12-02T03:18:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:47 2023-12-02T03:18:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:56 2023-12-02T03:18:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_38: 2023-12-02T03:18:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 8a609a08-21d0-4193-a27d-f581f4516454 2023-12-02T03:18:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486405,"remote_snapshot_timestamp":1701486405,"replay_state":"idle"} 2023-12-02T03:18:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.1 on smithi057 2023-12-02T03:18:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:57 2023-12-02T03:18:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.423 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:56 2023-12-02T03:18:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_39: 2023-12-02T03:18:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: c1ab0b04-68aa-4508-ad3f-25906784bf9e 2023-12-02T03:18:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486416,"remote_snapshot_timestamp":1701486416,"replay_state":"idle"} 2023-12-02T03:18:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.1 on smithi057 2023-12-02T03:18:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:57 2023-12-02T03:18:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:56 2023-12-02T03:18:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_4: 2023-12-02T03:18:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 730af9f3-f6b6-4cf2-9474-5684c31f584d 2023-12-02T03:18:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486626,"remote_snapshot_timestamp":1701486626,"replay_state":"idle"} 2023-12-02T03:18:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.1 on smithi057 2023-12-02T03:18:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:57 2023-12-02T03:18:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: down+stopped 2023-12-02T03:18:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: local image is primary 2023-12-02T03:18:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:12:50 2023-12-02T03:18:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_40: 2023-12-02T03:18:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: f12eeb37-6efd-4241-9808-5371a9611fae 2023-12-02T03:18:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486429,"remote_snapshot_timestamp":1701486429,"replay_state":"idle"} 2023-12-02T03:18:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.1 on smithi057 2023-12-02T03:18:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:57 2023-12-02T03:18:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:48 2023-12-02T03:18:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_41: 2023-12-02T03:18:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 5db3178e-6670-496c-bd76-373d30e83937 2023-12-02T03:18:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486441,"remote_snapshot_timestamp":1701486441,"replay_state":"idle"} 2023-12-02T03:18:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.0 on smithi057 2023-12-02T03:18:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:18:07 2023-12-02T03:18:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: down+stopped 2023-12-02T03:18:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: local image is primary 2023-12-02T03:18:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:12:50 2023-12-02T03:18:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_42: 2023-12-02T03:18:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 55cf3916-f72c-47f2-b9d9-b3f8b8496445 2023-12-02T03:18:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486448,"remote_snapshot_timestamp":1701486448,"replay_state":"idle"} 2023-12-02T03:18:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.3 on smithi057 2023-12-02T03:18:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:51 2023-12-02T03:18:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: down+stopped 2023-12-02T03:18:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: local image is primary 2023-12-02T03:18:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:12:50 2023-12-02T03:18:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_43: 2023-12-02T03:18:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 0c9ffe84-d24c-4f74-b087-b033f2cc4422 2023-12-02T03:18:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486461,"remote_snapshot_timestamp":1701486461,"replay_state":"idle"} 2023-12-02T03:18:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.3 on smithi057 2023-12-02T03:18:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:51 2023-12-02T03:18:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:56 2023-12-02T03:18:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_44: 2023-12-02T03:18:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 2e4177a2-c078-4cd9-b5d2-ee2477a02290 2023-12-02T03:18:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486472,"remote_snapshot_timestamp":1701486472,"replay_state":"idle"} 2023-12-02T03:18:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.3 on smithi057 2023-12-02T03:18:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:51 2023-12-02T03:18:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:48 2023-12-02T03:18:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_45: 2023-12-02T03:18:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: c46da9c3-6d9c-4389-a394-a73d56fb3cdf 2023-12-02T03:18:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486479,"remote_snapshot_timestamp":1701486479,"replay_state":"idle"} 2023-12-02T03:18:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.2 on smithi057 2023-12-02T03:18:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:47 2023-12-02T03:18:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:56 2023-12-02T03:18:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_46: 2023-12-02T03:18:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 0a3b7484-3d9a-411e-9ed7-b9ca249bd93a 2023-12-02T03:18:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486490,"remote_snapshot_timestamp":1701486490,"replay_state":"idle"} 2023-12-02T03:18:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.2 on smithi057 2023-12-02T03:18:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:47 2023-12-02T03:18:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:18:16 2023-12-02T03:18:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_47: 2023-12-02T03:18:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: be8f73e3-45b7-4b7e-8479-7fd35126d17d 2023-12-02T03:18:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486504,"remote_snapshot_timestamp":1701486504,"replay_state":"idle"} 2023-12-02T03:18:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.1 on smithi057 2023-12-02T03:18:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:57 2023-12-02T03:18:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:56 2023-12-02T03:18:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_48: 2023-12-02T03:18:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 4f77376e-57d2-4b09-b376-6129ff7390b9 2023-12-02T03:18:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486513,"remote_snapshot_timestamp":1701486513,"replay_state":"idle"} 2023-12-02T03:18:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.3 on smithi057 2023-12-02T03:18:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:51 2023-12-02T03:18:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:48 2023-12-02T03:18:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_49: 2023-12-02T03:18:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 92a00385-f1d8-4e86-abb7-0e1810b799c3 2023-12-02T03:18:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486519,"remote_snapshot_timestamp":1701486519,"replay_state":"idle"} 2023-12-02T03:18:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.2 on smithi057 2023-12-02T03:18:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:47 2023-12-02T03:18:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: down+stopped 2023-12-02T03:18:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: local image is primary 2023-12-02T03:18:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:12:50 2023-12-02T03:18:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_5: 2023-12-02T03:18:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 19ecaa0e-982b-435c-a3a7-d881b478cee6 2023-12-02T03:18:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486632,"remote_snapshot_timestamp":1701486632,"replay_state":"idle"} 2023-12-02T03:18:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.0 on smithi057 2023-12-02T03:18:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:18:07 2023-12-02T03:18:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: down+stopped 2023-12-02T03:18:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: local image is primary 2023-12-02T03:18:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:12:50 2023-12-02T03:18:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_50: 2023-12-02T03:18:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: f939d359-e3e3-4881-b4c7-c05064e62969 2023-12-02T03:18:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486534,"remote_snapshot_timestamp":1701486534,"replay_state":"idle"} 2023-12-02T03:18:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.2 on smithi057 2023-12-02T03:18:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:47 2023-12-02T03:18:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:48 2023-12-02T03:18:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_6: 2023-12-02T03:18:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: f1a9b266-fc3d-4246-8a57-fe1695af4b4d 2023-12-02T03:18:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":134205440.0,"last_snapshot_bytes":134205440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486638,"remote_snapshot_timestamp":1701486638,"replay_state":"idle"} 2023-12-02T03:18:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.2 on smithi057 2023-12-02T03:18:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:47 2023-12-02T03:18:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:56 2023-12-02T03:18:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_7: 2023-12-02T03:18:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: 14b5094b-8df2-410f-b1dd-ae91594e9848 2023-12-02T03:18:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133709824.0,"last_snapshot_bytes":133709824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486644,"remote_snapshot_timestamp":1701486644,"replay_state":"idle"} 2023-12-02T03:18:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.3 on smithi057 2023-12-02T03:18:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:51 2023-12-02T03:18:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:18:16 2023-12-02T03:18:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_8: 2023-12-02T03:18:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: f71e0710-9762-4b0b-8375-4f15e3c521f1 2023-12-02T03:18:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486650,"remote_snapshot_timestamp":1701486650,"replay_state":"idle"} 2023-12-02T03:18:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.1 on smithi057 2023-12-02T03:18:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:57 2023-12-02T03:18:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:48 2023-12-02T03:18:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_9: 2023-12-02T03:18:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: global_id: da41f19a-67cd-4518-a939-22e8fe4b6971 2023-12-02T03:18:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+replaying 2023-12-02T03:18:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701486656,"remote_snapshot_timestamp":1701486656,"replay_state":"idle"} 2023-12-02T03:18:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: service: mirror.2 on smithi057 2023-12-02T03:18:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:47 2023-12-02T03:18:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: peer_sites: 2023-12-02T03:18:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: name: cluster2 2023-12-02T03:18:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: state: up+starting_replay 2023-12-02T03:18:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: description: starting replay 2023-12-02T03:18:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: last_update: 2023-12-02 03:17:56 2023-12-02T03:18:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ rbd --cluster cluster1 -p mirror --namespace '' ls 2023-12-02T03:18:16.495 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_1 info' 2023-12-02T03:18:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_1 info 2023-12-02T03:18:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_1 2023-12-02T03:18:16.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_1': 2023-12-02T03:18:16.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:16.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:16.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:18:16.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 10e4cf0fd5b2 2023-12-02T03:18:16.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.10e4cf0fd5b2 2023-12-02T03:18:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 02:59:41 2023 2023-12-02T03:18:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:09:38 2023 2023-12-02T03:18:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 02:59:41 2023 2023-12-02T03:18:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 49c16484-3472-4671-8be8-03c1846564a3 2023-12-02T03:18:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_1 journal status 2023-12-02T03:18:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_1 journal status' 2023-12-02T03:18:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_1 2023-12-02T03:18:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_1 2023-12-02T03:18:16.788 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.788 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:16.788 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_1 snapshots 2023-12-02T03:18:16.789 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_1 snapshots' 2023-12-02T03:18:16.789 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_1 2023-12-02T03:18:16.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:16.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 118 snap 128 MiB Sat Dec 2 03:09:34 2023 user 2023-12-02T03:18:16.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 119 .mirror.non_primary.49c16484-3472-4671-8be8-03c1846564a3.7a0b5194-8e95-4931-8cd5-203ad2fc4581 128 MiB Sat Dec 2 03:09:35 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:119 copied) 2023-12-02T03:18:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_10 info' 2023-12-02T03:18:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_10 info 2023-12-02T03:18:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_10 2023-12-02T03:18:16.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_10': 2023-12-02T03:18:16.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:16.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:16.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:18:16.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 11b8b21d95bc 2023-12-02T03:18:16.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.11b8b21d95bc 2023-12-02T03:18:16.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:16.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:01:25 2023 2023-12-02T03:18:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:11:07 2023 2023-12-02T03:18:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:01:25 2023 2023-12-02T03:18:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: ddeae318-0d59-4bf2-b55e-a3f9934ab1b6 2023-12-02T03:18:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:16.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_10 journal status 2023-12-02T03:18:16.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:16.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_10 journal status' 2023-12-02T03:18:16.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_10 2023-12-02T03:18:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_10 2023-12-02T03:18:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_10 snapshots 2023-12-02T03:18:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_10 snapshots' 2023-12-02T03:18:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_10 2023-12-02T03:18:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 145 snap 128 MiB Sat Dec 2 03:11:03 2023 user 2023-12-02T03:18:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 146 .mirror.non_primary.ddeae318-0d59-4bf2-b55e-a3f9934ab1b6.7edabe7d-050f-47db-91a1-d4cbabe488e1 128 MiB Sat Dec 2 03:11:04 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:137 copied) 2023-12-02T03:18:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_11 info' 2023-12-02T03:18:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_11 info 2023-12-02T03:18:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_11 2023-12-02T03:18:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_11': 2023-12-02T03:18:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:18:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12115e9f77d7 2023-12-02T03:18:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12115e9f77d7 2023-12-02T03:18:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:01:38 2023 2023-12-02T03:18:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:11:15 2023 2023-12-02T03:18:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:01:38 2023 2023-12-02T03:18:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 258c1b03-48a5-42f2-8745-ac0283c8bc87 2023-12-02T03:18:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_11 journal status 2023-12-02T03:18:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_11 journal status' 2023-12-02T03:18:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_11 2023-12-02T03:18:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_11 2023-12-02T03:18:17.061 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_11 snapshots 2023-12-02T03:18:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_11 snapshots' 2023-12-02T03:18:17.063 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_11 2023-12-02T03:18:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 148 snap 128 MiB Sat Dec 2 03:11:11 2023 user 2023-12-02T03:18:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 149 .mirror.non_primary.258c1b03-48a5-42f2-8745-ac0283c8bc87.280e6eff-2ec0-429b-8289-29a465ff5cdb 128 MiB Sat Dec 2 03:11:12 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:139 copied) 2023-12-02T03:18:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_12 info' 2023-12-02T03:18:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_12 info 2023-12-02T03:18:17.108 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_12 2023-12-02T03:18:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_12': 2023-12-02T03:18:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:18:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12a110e3d821 2023-12-02T03:18:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12a110e3d821 2023-12-02T03:18:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:01:53 2023 2023-12-02T03:18:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:11:21 2023 2023-12-02T03:18:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:01:53 2023 2023-12-02T03:18:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: a30b9861-e056-4151-bcfe-e4a4f2e867dd 2023-12-02T03:18:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:17.153 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:17.153 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_12 journal status 2023-12-02T03:18:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_12 journal status' 2023-12-02T03:18:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_12 2023-12-02T03:18:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_12 2023-12-02T03:18:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_12 snapshots 2023-12-02T03:18:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_12 snapshots' 2023-12-02T03:18:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_12 2023-12-02T03:18:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 151 snap 128 MiB Sat Dec 2 03:11:18 2023 user 2023-12-02T03:18:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 152 .mirror.non_primary.a30b9861-e056-4151-bcfe-e4a4f2e867dd.9619e099-eb6f-499f-bf93-6853588150e5 128 MiB Sat Dec 2 03:11:19 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:141 copied) 2023-12-02T03:18:17.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:17.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:17.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:17.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_13 info 2023-12-02T03:18:17.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_13 info' 2023-12-02T03:18:17.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_13 2023-12-02T03:18:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_13': 2023-12-02T03:18:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:18:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 10e49ec39909 2023-12-02T03:18:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.10e49ec39909 2023-12-02T03:18:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:02:03 2023 2023-12-02T03:18:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:11:28 2023 2023-12-02T03:18:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:02:03 2023 2023-12-02T03:18:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: cdf57a21-3ee2-466a-9e94-c4dd4500ddd1 2023-12-02T03:18:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:17.292 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_13 journal status 2023-12-02T03:18:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_13 journal status' 2023-12-02T03:18:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_13 2023-12-02T03:18:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_13 2023-12-02T03:18:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_13 snapshots 2023-12-02T03:18:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_13 snapshots' 2023-12-02T03:18:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_13 2023-12-02T03:18:17.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:17.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 154 snap 128 MiB Sat Dec 2 03:11:24 2023 user 2023-12-02T03:18:17.382 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 155 .mirror.non_primary.cdf57a21-3ee2-466a-9e94-c4dd4500ddd1.95e1548b-234a-415d-bba3-455cb1c76704 128 MiB Sat Dec 2 03:11:25 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:143 copied) 2023-12-02T03:18:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_14 info 2023-12-02T03:18:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_14 info' 2023-12-02T03:18:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_14 2023-12-02T03:18:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_14': 2023-12-02T03:18:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:18:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 11b838851b3 2023-12-02T03:18:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.11b838851b3 2023-12-02T03:18:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:02:16 2023 2023-12-02T03:18:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:11:36 2023 2023-12-02T03:18:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:02:16 2023 2023-12-02T03:18:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 1d600ef5-05fa-4ec8-bb36-7d891860097c 2023-12-02T03:18:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_14 journal status 2023-12-02T03:18:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_14 journal status' 2023-12-02T03:18:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_14 2023-12-02T03:18:17.473 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_14 2023-12-02T03:18:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_14 snapshots 2023-12-02T03:18:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_14 snapshots' 2023-12-02T03:18:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_14 2023-12-02T03:18:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 157 snap 128 MiB Sat Dec 2 03:11:32 2023 user 2023-12-02T03:18:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 158 .mirror.non_primary.1d600ef5-05fa-4ec8-bb36-7d891860097c.7e576452-f12f-4fb2-be42-5c1a52b7b806 128 MiB Sat Dec 2 03:11:33 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:145 copied) 2023-12-02T03:18:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_15 info 2023-12-02T03:18:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_15 info' 2023-12-02T03:18:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_15 2023-12-02T03:18:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_15': 2023-12-02T03:18:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:18:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 1211da7dd79c 2023-12-02T03:18:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.1211da7dd79c 2023-12-02T03:18:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:02:26 2023 2023-12-02T03:18:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:11:42 2023 2023-12-02T03:18:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:02:26 2023 2023-12-02T03:18:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 5a34854e-65bf-43d7-9d2c-e0481cc5a159 2023-12-02T03:18:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_15 journal status 2023-12-02T03:18:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_15 journal status' 2023-12-02T03:18:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_15 2023-12-02T03:18:17.609 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_15 2023-12-02T03:18:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_15 snapshots' 2023-12-02T03:18:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_15 snapshots 2023-12-02T03:18:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_15 2023-12-02T03:18:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 160 snap 128 MiB Sat Dec 2 03:11:39 2023 user 2023-12-02T03:18:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 161 .mirror.non_primary.5a34854e-65bf-43d7-9d2c-e0481cc5a159.73004f99-b039-43c8-ad30-706bf5625f4a 128 MiB Sat Dec 2 03:11:40 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:147 copied) 2023-12-02T03:18:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_16 info 2023-12-02T03:18:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_16 info' 2023-12-02T03:18:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_16 2023-12-02T03:18:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_16': 2023-12-02T03:18:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:18:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12b1e1c6312a 2023-12-02T03:18:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12b1e1c6312a 2023-12-02T03:18:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:02:37 2023 2023-12-02T03:18:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:11:50 2023 2023-12-02T03:18:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:02:37 2023 2023-12-02T03:18:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 807f0ce9-67ef-4e3a-987e-bfe34f97a832 2023-12-02T03:18:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_16 journal status 2023-12-02T03:18:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_16 journal status' 2023-12-02T03:18:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_16 2023-12-02T03:18:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_16 2023-12-02T03:18:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_16 snapshots 2023-12-02T03:18:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_16 snapshots' 2023-12-02T03:18:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_16 2023-12-02T03:18:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 163 snap 128 MiB Sat Dec 2 03:11:46 2023 user 2023-12-02T03:18:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 164 .mirror.non_primary.807f0ce9-67ef-4e3a-987e-bfe34f97a832.7b35eeaa-f244-4897-9c29-b7d3528cebd0 128 MiB Sat Dec 2 03:11:47 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:149 copied) 2023-12-02T03:18:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_17 info 2023-12-02T03:18:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_17 info' 2023-12-02T03:18:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_17 2023-12-02T03:18:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_17': 2023-12-02T03:18:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:18:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12b1cd66238 2023-12-02T03:18:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12b1cd66238 2023-12-02T03:18:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:02:51 2023 2023-12-02T03:18:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:11:56 2023 2023-12-02T03:18:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:02:51 2023 2023-12-02T03:18:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 1f4ea1ff-7710-42b2-9726-194a70ed8f1e 2023-12-02T03:18:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:17.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_17 journal status 2023-12-02T03:18:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_17 journal status' 2023-12-02T03:18:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_17 2023-12-02T03:18:17.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_17 2023-12-02T03:18:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_17 snapshots 2023-12-02T03:18:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_17 snapshots' 2023-12-02T03:18:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_17 2023-12-02T03:18:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 166 snap 128 MiB Sat Dec 2 03:11:53 2023 user 2023-12-02T03:18:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 167 .mirror.non_primary.1f4ea1ff-7710-42b2-9726-194a70ed8f1e.b2c9e418-150f-4438-9c76-33debbf94b39 128 MiB Sat Dec 2 03:11:54 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:151 copied) 2023-12-02T03:18:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_18 info 2023-12-02T03:18:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_18 info' 2023-12-02T03:18:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_18 2023-12-02T03:18:17.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_18': 2023-12-02T03:18:17.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:17.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:17.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:18:17.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12118d51a868 2023-12-02T03:18:17.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12118d51a868 2023-12-02T03:18:17.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:03:00 2023 2023-12-02T03:18:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:12:02 2023 2023-12-02T03:18:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:03:00 2023 2023-12-02T03:18:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: b8d1a79d-fa19-4b64-ba6b-5a4658335ba3 2023-12-02T03:18:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:17.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:17.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_18 journal status 2023-12-02T03:18:17.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_18 journal status' 2023-12-02T03:18:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_18 2023-12-02T03:18:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_18 2023-12-02T03:18:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_18 snapshots 2023-12-02T03:18:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_18 snapshots' 2023-12-02T03:18:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_18 2023-12-02T03:18:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 169 snap 128 MiB Sat Dec 2 03:11:59 2023 user 2023-12-02T03:18:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 170 .mirror.non_primary.b8d1a79d-fa19-4b64-ba6b-5a4658335ba3.f7df8e9d-d4e5-4981-9f3e-71d8050d4f28 128 MiB Sat Dec 2 03:12:00 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:153 copied) 2023-12-02T03:18:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:18.067 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:18.067 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:18.067 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_19 info 2023-12-02T03:18:18.067 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_19 info' 2023-12-02T03:18:18.067 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_19 2023-12-02T03:18:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_19': 2023-12-02T03:18:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:18:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12a1f60a4dad 2023-12-02T03:18:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12a1f60a4dad 2023-12-02T03:18:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:03:13 2023 2023-12-02T03:18:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:12:10 2023 2023-12-02T03:18:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:03:13 2023 2023-12-02T03:18:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: faedaa7c-e337-491d-80ec-3cfa45a9493c 2023-12-02T03:18:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_19 journal status 2023-12-02T03:18:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_19 journal status' 2023-12-02T03:18:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_19 2023-12-02T03:18:18.156 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_19 2023-12-02T03:18:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_19 snapshots 2023-12-02T03:18:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_19 snapshots' 2023-12-02T03:18:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_19 2023-12-02T03:18:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 172 snap 128 MiB Sat Dec 2 03:12:06 2023 user 2023-12-02T03:18:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 173 .mirror.non_primary.faedaa7c-e337-491d-80ec-3cfa45a9493c.11af6503-5baa-4804-8729-09c5dcb2852b 128 MiB Sat Dec 2 03:12:07 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:155 copied) 2023-12-02T03:18:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_2 info' 2023-12-02T03:18:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_2 info 2023-12-02T03:18:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_2 2023-12-02T03:18:18.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_2': 2023-12-02T03:18:18.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:18.247 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:18:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 115ef4dd5c1d 2023-12-02T03:18:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.115ef4dd5c1d 2023-12-02T03:18:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 02:59:54 2023 2023-12-02T03:18:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:09:44 2023 2023-12-02T03:18:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 02:59:54 2023 2023-12-02T03:18:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 09e06fe9-a175-4009-952b-403e63e1ac10 2023-12-02T03:18:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_2 journal status 2023-12-02T03:18:18.251 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_2 journal status' 2023-12-02T03:18:18.251 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_2 2023-12-02T03:18:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_2 2023-12-02T03:18:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_2 snapshots 2023-12-02T03:18:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_2 snapshots' 2023-12-02T03:18:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_2 2023-12-02T03:18:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 121 snap 128 MiB Sat Dec 2 03:09:41 2023 user 2023-12-02T03:18:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 122 .mirror.non_primary.09e06fe9-a175-4009-952b-403e63e1ac10.e2891bcb-14b1-4793-9758-e5677a7788dd 128 MiB Sat Dec 2 03:09:42 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:121 copied) 2023-12-02T03:18:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_20 info 2023-12-02T03:18:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_20 info' 2023-12-02T03:18:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_20 2023-12-02T03:18:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_20': 2023-12-02T03:18:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:18:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12b1f96fe282 2023-12-02T03:18:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12b1f96fe282 2023-12-02T03:18:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:03:25 2023 2023-12-02T03:18:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:12:17 2023 2023-12-02T03:18:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:03:25 2023 2023-12-02T03:18:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 8117a00a-84c1-4b06-96bd-60840c20cce4 2023-12-02T03:18:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_20 journal status 2023-12-02T03:18:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_20 journal status' 2023-12-02T03:18:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_20 2023-12-02T03:18:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_20 2023-12-02T03:18:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_20 snapshots 2023-12-02T03:18:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_20 snapshots' 2023-12-02T03:18:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_20 2023-12-02T03:18:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 175 snap 128 MiB Sat Dec 2 03:12:13 2023 user 2023-12-02T03:18:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 176 .mirror.non_primary.8117a00a-84c1-4b06-96bd-60840c20cce4.1ee710ec-be8d-44fc-b644-45056f8abd87 128 MiB Sat Dec 2 03:12:14 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:157 copied) 2023-12-02T03:18:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_21 info 2023-12-02T03:18:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_21 info' 2023-12-02T03:18:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_21 2023-12-02T03:18:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_21': 2023-12-02T03:18:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:18:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 11b8936b5756 2023-12-02T03:18:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.11b8936b5756 2023-12-02T03:18:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:03:32 2023 2023-12-02T03:18:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:12:23 2023 2023-12-02T03:18:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:03:32 2023 2023-12-02T03:18:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: b629353c-89f1-46c0-b5ff-f652e93d5cb3 2023-12-02T03:18:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:18.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:18.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_21 journal status 2023-12-02T03:18:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_21 journal status' 2023-12-02T03:18:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_21 2023-12-02T03:18:18.571 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_21 2023-12-02T03:18:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_21 snapshots 2023-12-02T03:18:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_21 snapshots' 2023-12-02T03:18:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_21 2023-12-02T03:18:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 178 snap 128 MiB Sat Dec 2 03:12:20 2023 user 2023-12-02T03:18:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 179 .mirror.non_primary.b629353c-89f1-46c0-b5ff-f652e93d5cb3.3a184bb8-7a9b-446e-b057-e4192d68978c 128 MiB Sat Dec 2 03:12:21 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:159 copied) 2023-12-02T03:18:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_22 info 2023-12-02T03:18:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_22 info' 2023-12-02T03:18:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_22 2023-12-02T03:18:18.665 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_22': 2023-12-02T03:18:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:18:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12b1ed8e617c 2023-12-02T03:18:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12b1ed8e617c 2023-12-02T03:18:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:04:06 2023 2023-12-02T03:18:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:12:29 2023 2023-12-02T03:18:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:04:06 2023 2023-12-02T03:18:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 8ef4933a-3981-486a-8a90-10311d67bf00 2023-12-02T03:18:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:18.668 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:18.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:18.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_22 journal status 2023-12-02T03:18:18.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_22 journal status' 2023-12-02T03:18:18.669 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_22 2023-12-02T03:18:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_22 2023-12-02T03:18:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_22 snapshots' 2023-12-02T03:18:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_22 snapshots 2023-12-02T03:18:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_22 2023-12-02T03:18:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:18.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 181 snap 128 MiB Sat Dec 2 03:12:26 2023 user 2023-12-02T03:18:18.760 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 182 .mirror.non_primary.8ef4933a-3981-486a-8a90-10311d67bf00.c61fc19a-a885-43bf-9b92-4e57224b498e 128 MiB Sat Dec 2 03:12:27 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:161 copied) 2023-12-02T03:18:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:18.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:18.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:18.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_23 info 2023-12-02T03:18:18.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_23 info' 2023-12-02T03:18:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_23 2023-12-02T03:18:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_23': 2023-12-02T03:18:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:18:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12c0b00e005 2023-12-02T03:18:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12c0b00e005 2023-12-02T03:18:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:03:54 2023 2023-12-02T03:18:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:12:35 2023 2023-12-02T03:18:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:03:54 2023 2023-12-02T03:18:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 8696705f-855b-402d-afd6-1c4d55e06c91 2023-12-02T03:18:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_23 journal status 2023-12-02T03:18:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_23 journal status' 2023-12-02T03:18:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_23 2023-12-02T03:18:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_23 2023-12-02T03:18:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_23 snapshots 2023-12-02T03:18:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_23 snapshots' 2023-12-02T03:18:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_23 2023-12-02T03:18:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 184 snap 128 MiB Sat Dec 2 03:12:32 2023 user 2023-12-02T03:18:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 185 .mirror.non_primary.8696705f-855b-402d-afd6-1c4d55e06c91.87ae4ec3-d5b5-447e-a4d6-2e01a7623571 128 MiB Sat Dec 2 03:12:33 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:163 copied) 2023-12-02T03:18:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_24 info 2023-12-02T03:18:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_24 info' 2023-12-02T03:18:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_24 2023-12-02T03:18:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_24': 2023-12-02T03:18:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:18:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12c05c91ef70 2023-12-02T03:18:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12c05c91ef70 2023-12-02T03:18:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:04:02 2023 2023-12-02T03:18:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:13:06 2023 2023-12-02T03:18:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:04:02 2023 2023-12-02T03:18:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 9b3405db-a7dc-43f6-8462-f7325ff11d39 2023-12-02T03:18:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_24 journal status 2023-12-02T03:18:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_24 journal status' 2023-12-02T03:18:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_24 2023-12-02T03:18:18.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_24 2023-12-02T03:18:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_24 snapshots 2023-12-02T03:18:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_24 snapshots' 2023-12-02T03:18:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_24 2023-12-02T03:18:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 187 snap 128 MiB Sat Dec 2 03:13:03 2023 user 2023-12-02T03:18:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 188 .mirror.non_primary.9b3405db-a7dc-43f6-8462-f7325ff11d39.b43da295-ad07-4d92-b7ec-7ad3de01a424 128 MiB Sat Dec 2 03:13:04 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:165 copied) 2023-12-02T03:18:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_25 info 2023-12-02T03:18:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_25 info' 2023-12-02T03:18:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_25 2023-12-02T03:18:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_25': 2023-12-02T03:18:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:18:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 11b8c0e8dff8 2023-12-02T03:18:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.11b8c0e8dff8 2023-12-02T03:18:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:04:12 2023 2023-12-02T03:18:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:13:12 2023 2023-12-02T03:18:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:04:12 2023 2023-12-02T03:18:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 26af4e3d-22a4-46d1-af96-3f3836056461 2023-12-02T03:18:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_25 journal status 2023-12-02T03:18:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_25 journal status' 2023-12-02T03:18:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_25 2023-12-02T03:18:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_25 2023-12-02T03:18:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_25 snapshots 2023-12-02T03:18:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_25 snapshots' 2023-12-02T03:18:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_25 2023-12-02T03:18:19.171 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:19.171 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 190 snap 128 MiB Sat Dec 2 03:13:09 2023 user 2023-12-02T03:18:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 191 .mirror.non_primary.26af4e3d-22a4-46d1-af96-3f3836056461.71ca59df-8c60-4382-b5d3-40f17e5d7903 128 MiB Sat Dec 2 03:13:10 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:167 copied) 2023-12-02T03:18:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_26 info' 2023-12-02T03:18:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_26 info 2023-12-02T03:18:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_26 2023-12-02T03:18:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_26': 2023-12-02T03:18:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:19.218 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:18:19.218 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12a1f1817ac 2023-12-02T03:18:19.218 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12a1f1817ac 2023-12-02T03:18:19.218 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:19.218 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:19.218 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:04:23 2023 2023-12-02T03:18:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:04:23 2023 2023-12-02T03:18:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:04:23 2023 2023-12-02T03:18:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 7dc110f7-a667-4d4c-9e52-4c6fe497aadb 2023-12-02T03:18:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_26 journal status' 2023-12-02T03:18:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_26 journal status 2023-12-02T03:18:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_26 2023-12-02T03:18:19.262 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_26 2023-12-02T03:18:19.264 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:19.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:19.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_26 snapshots 2023-12-02T03:18:19.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_26 snapshots' 2023-12-02T03:18:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_26 2023-12-02T03:18:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 93 .mirror.non_primary.7dc110f7-a667-4d4c-9e52-4c6fe497aadb.72cacd00-a4f8-465d-9a1a-7734766911a6 128 MiB Sat Dec 2 03:04:24 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:93 copied) 2023-12-02T03:18:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_27 info 2023-12-02T03:18:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_27 info' 2023-12-02T03:18:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_27 2023-12-02T03:18:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_27': 2023-12-02T03:18:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:18:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12b1f9d98bb6 2023-12-02T03:18:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12b1f9d98bb6 2023-12-02T03:18:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:04:32 2023 2023-12-02T03:18:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:04:32 2023 2023-12-02T03:18:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:04:32 2023 2023-12-02T03:18:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 453a5e01-39a4-415e-bf21-d59460038eff 2023-12-02T03:18:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:19.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:19.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_27 journal status 2023-12-02T03:18:19.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:19.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_27 journal status' 2023-12-02T03:18:19.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_27 2023-12-02T03:18:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_27 2023-12-02T03:18:19.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:19.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_27 snapshots 2023-12-02T03:18:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_27 snapshots' 2023-12-02T03:18:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_27 2023-12-02T03:18:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 94 .mirror.non_primary.453a5e01-39a4-415e-bf21-d59460038eff.10cdd0cf-5604-44b1-9c0e-f6894ee2e10c 128 MiB Sat Dec 2 03:04:32 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:94 copied) 2023-12-02T03:18:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_28 info 2023-12-02T03:18:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_28 info' 2023-12-02T03:18:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_28 2023-12-02T03:18:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_28': 2023-12-02T03:18:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:18:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12c0d858d532 2023-12-02T03:18:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12c0d858d532 2023-12-02T03:18:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:04:45 2023 2023-12-02T03:18:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:04:45 2023 2023-12-02T03:18:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:04:45 2023 2023-12-02T03:18:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: e7466612-311c-416f-a7f9-bfa20e74a575 2023-12-02T03:18:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_28 journal status 2023-12-02T03:18:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_28 journal status' 2023-12-02T03:18:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_28 2023-12-02T03:18:19.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_28 2023-12-02T03:18:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_28 snapshots 2023-12-02T03:18:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_28 snapshots' 2023-12-02T03:18:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_28 2023-12-02T03:18:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 95 .mirror.non_primary.e7466612-311c-416f-a7f9-bfa20e74a575.07145ccf-7fb4-4bb2-83f0-0fa45100d6f3 128 MiB Sat Dec 2 03:04:46 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:95 copied) 2023-12-02T03:18:19.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:19.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:19.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:19.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_29 info 2023-12-02T03:18:19.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_29 info' 2023-12-02T03:18:19.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_29 2023-12-02T03:18:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_29': 2023-12-02T03:18:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:18:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 11b82ee336bb 2023-12-02T03:18:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.11b82ee336bb 2023-12-02T03:18:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:04:56 2023 2023-12-02T03:18:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:04:56 2023 2023-12-02T03:18:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:04:56 2023 2023-12-02T03:18:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 3d00c370-35ca-4837-a707-0aa3e5b4c7f4 2023-12-02T03:18:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_29 journal status 2023-12-02T03:18:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_29 journal status' 2023-12-02T03:18:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_29 2023-12-02T03:18:19.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_29 2023-12-02T03:18:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_29 snapshots' 2023-12-02T03:18:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_29 snapshots 2023-12-02T03:18:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_29 2023-12-02T03:18:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 96 .mirror.non_primary.3d00c370-35ca-4837-a707-0aa3e5b4c7f4.672ae72c-d051-4f9a-a038-1b9c0c2ce699 128 MiB Sat Dec 2 03:04:56 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:96 copied) 2023-12-02T03:18:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_3 info 2023-12-02T03:18:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_3 info' 2023-12-02T03:18:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_3 2023-12-02T03:18:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_3': 2023-12-02T03:18:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:18:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 11b8c106d4d5 2023-12-02T03:18:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.11b8c106d4d5 2023-12-02T03:18:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:00:05 2023 2023-12-02T03:18:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:09:50 2023 2023-12-02T03:18:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:00:05 2023 2023-12-02T03:18:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: a35ab54d-17a7-446d-984d-02da30173d31 2023-12-02T03:18:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_3 journal status 2023-12-02T03:18:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_3 journal status' 2023-12-02T03:18:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_3 2023-12-02T03:18:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_3 2023-12-02T03:18:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_3 snapshots 2023-12-02T03:18:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_3 snapshots' 2023-12-02T03:18:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_3 2023-12-02T03:18:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 124 snap 128 MiB Sat Dec 2 03:09:47 2023 user 2023-12-02T03:18:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 125 .mirror.non_primary.a35ab54d-17a7-446d-984d-02da30173d31.38a6b3f1-8fc2-4a09-8392-12047bc1633d 128 MiB Sat Dec 2 03:09:48 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:123 copied) 2023-12-02T03:18:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_30 info 2023-12-02T03:18:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_30 info' 2023-12-02T03:18:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_30 2023-12-02T03:18:19.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_30': 2023-12-02T03:18:19.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:19.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:19.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:18:19.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12a14a2ab28c 2023-12-02T03:18:19.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12a14a2ab28c 2023-12-02T03:18:19.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:19.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:19.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:19.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:19.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:05:09 2023 2023-12-02T03:18:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:05:09 2023 2023-12-02T03:18:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:05:09 2023 2023-12-02T03:18:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 03c2b3e5-9969-4a06-b152-32b7163be7b1 2023-12-02T03:18:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_30 journal status 2023-12-02T03:18:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_30 journal status' 2023-12-02T03:18:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_30 2023-12-02T03:18:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_30 2023-12-02T03:18:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_30 snapshots 2023-12-02T03:18:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_30 snapshots' 2023-12-02T03:18:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_30 2023-12-02T03:18:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 97 .mirror.non_primary.03c2b3e5-9969-4a06-b152-32b7163be7b1.cce75e91-f23f-45d4-8344-76fcf7771e52 128 MiB Sat Dec 2 03:05:10 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:97 copied) 2023-12-02T03:18:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_31 info 2023-12-02T03:18:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_31 info' 2023-12-02T03:18:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_31 2023-12-02T03:18:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_31': 2023-12-02T03:18:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:18:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12b1c1ffcbda 2023-12-02T03:18:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12b1c1ffcbda 2023-12-02T03:18:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:05:15 2023 2023-12-02T03:18:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:05:15 2023 2023-12-02T03:18:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:05:15 2023 2023-12-02T03:18:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: ffc4290c-3b4c-45f6-a30f-c93f3cb26275 2023-12-02T03:18:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_31 journal status 2023-12-02T03:18:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_31 journal status' 2023-12-02T03:18:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_31 2023-12-02T03:18:20.088 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_31 2023-12-02T03:18:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_31 snapshots 2023-12-02T03:18:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_31 snapshots' 2023-12-02T03:18:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_31 2023-12-02T03:18:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 98 .mirror.non_primary.ffc4290c-3b4c-45f6-a30f-c93f3cb26275.5b2a1bf7-692e-43e2-9c37-845fc792957c 128 MiB Sat Dec 2 03:05:16 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:98 copied) 2023-12-02T03:18:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_32 info 2023-12-02T03:18:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_32 info' 2023-12-02T03:18:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_32 2023-12-02T03:18:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_32': 2023-12-02T03:18:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:18:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12c02536d968 2023-12-02T03:18:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12c02536d968 2023-12-02T03:18:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:05:30 2023 2023-12-02T03:18:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:05:30 2023 2023-12-02T03:18:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:05:30 2023 2023-12-02T03:18:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: d4ceac3d-c782-416e-a7e1-be445a43ae35 2023-12-02T03:18:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_32 journal status 2023-12-02T03:18:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_32 journal status' 2023-12-02T03:18:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_32 2023-12-02T03:18:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_32 2023-12-02T03:18:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_32 snapshots 2023-12-02T03:18:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_32 snapshots' 2023-12-02T03:18:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_32 2023-12-02T03:18:20.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi057.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-02T03:18:20.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi057.stderr:daemon-helper: command failed with exit status 1 2023-12-02T03:18:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 99 .mirror.non_primary.d4ceac3d-c782-416e-a7e1-be445a43ae35.3d7a62af-3697-45e2-b9a4-9c99c5120c31 128 MiB Sat Dec 2 03:05:31 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:99 copied) 2023-12-02T03:18:20.274 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:20.274 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:20.274 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_33 info 2023-12-02T03:18:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_33 info' 2023-12-02T03:18:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_33 2023-12-02T03:18:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_33': 2023-12-02T03:18:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:18:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 11b821202083 2023-12-02T03:18:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.11b821202083 2023-12-02T03:18:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:05:41 2023 2023-12-02T03:18:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:05:41 2023 2023-12-02T03:18:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:05:41 2023 2023-12-02T03:18:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 56d75e57-f434-4172-81f2-851edf5c5eb5 2023-12-02T03:18:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_33 journal status 2023-12-02T03:18:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_33 journal status' 2023-12-02T03:18:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_33 2023-12-02T03:18:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_33 2023-12-02T03:18:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_33 snapshots 2023-12-02T03:18:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_33 snapshots' 2023-12-02T03:18:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_33 2023-12-02T03:18:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 100 .mirror.non_primary.56d75e57-f434-4172-81f2-851edf5c5eb5.9c64065b-7100-4e9b-82ad-c9d9039df819 128 MiB Sat Dec 2 03:05:42 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:100 copied) 2023-12-02T03:18:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_34 info' 2023-12-02T03:18:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_34 info 2023-12-02T03:18:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_34 2023-12-02T03:18:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_34': 2023-12-02T03:18:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:18:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12a125389e21 2023-12-02T03:18:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12a125389e21 2023-12-02T03:18:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:05:55 2023 2023-12-02T03:18:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:05:55 2023 2023-12-02T03:18:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:05:55 2023 2023-12-02T03:18:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: e2407e91-d52b-4a70-b3dd-c23362ba73c5 2023-12-02T03:18:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:20.455 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:20.455 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_34 journal status 2023-12-02T03:18:20.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:20.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_34 journal status' 2023-12-02T03:18:20.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_34 2023-12-02T03:18:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_34 2023-12-02T03:18:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_34 snapshots 2023-12-02T03:18:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_34 snapshots' 2023-12-02T03:18:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_34 2023-12-02T03:18:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:20.544 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 101 .mirror.non_primary.e2407e91-d52b-4a70-b3dd-c23362ba73c5.6f7ec449-0c81-4d41-b82d-59f89f3bf9a0 128 MiB Sat Dec 2 03:05:55 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:101 copied) 2023-12-02T03:18:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_35 info 2023-12-02T03:18:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_35 info' 2023-12-02T03:18:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_35 2023-12-02T03:18:20.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_35': 2023-12-02T03:18:20.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:20.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:18:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12b1c1e9bd9e 2023-12-02T03:18:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12b1c1e9bd9e 2023-12-02T03:18:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:06:08 2023 2023-12-02T03:18:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:06:08 2023 2023-12-02T03:18:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:06:08 2023 2023-12-02T03:18:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 9802897e-7ccb-4bb2-8ab0-cedc09b0044e 2023-12-02T03:18:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_35 journal status 2023-12-02T03:18:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_35 journal status' 2023-12-02T03:18:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_35 2023-12-02T03:18:20.635 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_35 2023-12-02T03:18:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_35 snapshots 2023-12-02T03:18:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_35 snapshots' 2023-12-02T03:18:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_35 2023-12-02T03:18:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 102 .mirror.non_primary.9802897e-7ccb-4bb2-8ab0-cedc09b0044e.91a7eacc-1d16-4d5e-a0ae-99df048fad09 128 MiB Sat Dec 2 03:06:09 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:102 copied) 2023-12-02T03:18:20.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:20.683 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_36 info 2023-12-02T03:18:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_36 info' 2023-12-02T03:18:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_36 2023-12-02T03:18:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_36': 2023-12-02T03:18:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:18:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12c0675f03b2 2023-12-02T03:18:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12c0675f03b2 2023-12-02T03:18:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:06:19 2023 2023-12-02T03:18:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:06:19 2023 2023-12-02T03:18:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:06:19 2023 2023-12-02T03:18:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 5c374561-6563-424e-bb48-6c5014bde1fc 2023-12-02T03:18:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_36 journal status 2023-12-02T03:18:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_36 journal status' 2023-12-02T03:18:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_36 2023-12-02T03:18:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_36 2023-12-02T03:18:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_36 snapshots 2023-12-02T03:18:20.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_36 snapshots' 2023-12-02T03:18:20.776 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_36 2023-12-02T03:18:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 103 .mirror.non_primary.5c374561-6563-424e-bb48-6c5014bde1fc.05d96049-ba9a-473f-86fc-bc5e10f462a1 128 MiB Sat Dec 2 03:06:20 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:103 copied) 2023-12-02T03:18:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_37 info' 2023-12-02T03:18:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_37 info 2023-12-02T03:18:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_37 2023-12-02T03:18:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_37': 2023-12-02T03:18:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:18:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 11b8c21eca37 2023-12-02T03:18:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.11b8c21eca37 2023-12-02T03:18:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:06:33 2023 2023-12-02T03:18:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:06:33 2023 2023-12-02T03:18:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:06:33 2023 2023-12-02T03:18:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 9333a204-121f-424e-849b-e6aec78f4224 2023-12-02T03:18:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_37 journal status 2023-12-02T03:18:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_37 journal status' 2023-12-02T03:18:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_37 2023-12-02T03:18:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_37 2023-12-02T03:18:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_37 snapshots 2023-12-02T03:18:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_37 snapshots' 2023-12-02T03:18:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_37 2023-12-02T03:18:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 104 .mirror.non_primary.9333a204-121f-424e-849b-e6aec78f4224.d88a43e3-6e90-4c4f-85bf-2d4f98ae38f3 128 MiB Sat Dec 2 03:06:34 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:104 copied) 2023-12-02T03:18:20.955 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_38 info' 2023-12-02T03:18:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_38 info 2023-12-02T03:18:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_38 2023-12-02T03:18:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_38': 2023-12-02T03:18:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:18:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12a124141a97 2023-12-02T03:18:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12a124141a97 2023-12-02T03:18:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:06:47 2023 2023-12-02T03:18:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:06:47 2023 2023-12-02T03:18:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:06:47 2023 2023-12-02T03:18:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 8a609a08-21d0-4193-a27d-f581f4516454 2023-12-02T03:18:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:21.000 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:21.000 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_38 journal status 2023-12-02T03:18:21.000 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_38 journal status' 2023-12-02T03:18:21.000 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_38 2023-12-02T03:18:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_38 2023-12-02T03:18:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_38 snapshots 2023-12-02T03:18:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_38 snapshots' 2023-12-02T03:18:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_38 2023-12-02T03:18:21.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:21.087 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 105 .mirror.non_primary.8a609a08-21d0-4193-a27d-f581f4516454.f3dfa4de-9d89-4f53-8951-91a7a7eae6b2 128 MiB Sat Dec 2 03:06:48 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:105 copied) 2023-12-02T03:18:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_39 info 2023-12-02T03:18:21.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_39 info' 2023-12-02T03:18:21.091 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_39 2023-12-02T03:18:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_39': 2023-12-02T03:18:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:18:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12b14109f85c 2023-12-02T03:18:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12b14109f85c 2023-12-02T03:18:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:06:58 2023 2023-12-02T03:18:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:06:58 2023 2023-12-02T03:18:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:06:58 2023 2023-12-02T03:18:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: c1ab0b04-68aa-4508-ad3f-25906784bf9e 2023-12-02T03:18:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_39 journal status 2023-12-02T03:18:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_39 journal status' 2023-12-02T03:18:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_39 2023-12-02T03:18:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_39 2023-12-02T03:18:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_39 snapshots 2023-12-02T03:18:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_39 snapshots' 2023-12-02T03:18:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_39 2023-12-02T03:18:21.221 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:21.222 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 106 .mirror.non_primary.c1ab0b04-68aa-4508-ad3f-25906784bf9e.50abbd5f-8368-4900-a976-dba301099d40 128 MiB Sat Dec 2 03:06:59 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:106 copied) 2023-12-02T03:18:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_4 info 2023-12-02T03:18:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_4 info' 2023-12-02T03:18:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_4 2023-12-02T03:18:21.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_4': 2023-12-02T03:18:21.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:21.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:18:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 121175fc59c5 2023-12-02T03:18:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.121175fc59c5 2023-12-02T03:18:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:00:15 2023 2023-12-02T03:18:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:10:29 2023 2023-12-02T03:18:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:00:15 2023 2023-12-02T03:18:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 730af9f3-f6b6-4cf2-9474-5684c31f584d 2023-12-02T03:18:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:21.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:21.269 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_4 journal status 2023-12-02T03:18:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_4 journal status' 2023-12-02T03:18:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_4 2023-12-02T03:18:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_4 2023-12-02T03:18:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_4 snapshots' 2023-12-02T03:18:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_4 snapshots 2023-12-02T03:18:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_4 2023-12-02T03:18:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 127 snap 128 MiB Sat Dec 2 03:10:26 2023 user 2023-12-02T03:18:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 128 .mirror.non_primary.730af9f3-f6b6-4cf2-9474-5684c31f584d.1e4bc414-f72d-4bfd-b82f-69836408f19d 128 MiB Sat Dec 2 03:10:27 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:125 copied) 2023-12-02T03:18:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_40 info 2023-12-02T03:18:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_40 info' 2023-12-02T03:18:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_40 2023-12-02T03:18:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_40': 2023-12-02T03:18:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:18:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12c041305f5d 2023-12-02T03:18:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12c041305f5d 2023-12-02T03:18:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:07:11 2023 2023-12-02T03:18:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:07:11 2023 2023-12-02T03:18:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:07:11 2023 2023-12-02T03:18:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: f12eeb37-6efd-4241-9808-5371a9611fae 2023-12-02T03:18:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_40 journal status 2023-12-02T03:18:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_40 journal status' 2023-12-02T03:18:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_40 2023-12-02T03:18:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_40 2023-12-02T03:18:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:21.449 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:21.449 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_40 snapshots 2023-12-02T03:18:21.449 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_40 snapshots' 2023-12-02T03:18:21.450 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_40 2023-12-02T03:18:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 107 .mirror.non_primary.f12eeb37-6efd-4241-9808-5371a9611fae.577b0dc7-4c32-4883-b25a-3830ab305600 128 MiB Sat Dec 2 03:07:12 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:107 copied) 2023-12-02T03:18:21.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:21.495 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:21.495 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:21.495 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_41 info 2023-12-02T03:18:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_41 info' 2023-12-02T03:18:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_41 2023-12-02T03:18:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_41': 2023-12-02T03:18:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:18:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 11b8bed4837c 2023-12-02T03:18:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.11b8bed4837c 2023-12-02T03:18:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:07:23 2023 2023-12-02T03:18:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:07:23 2023 2023-12-02T03:18:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:07:23 2023 2023-12-02T03:18:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 5db3178e-6670-496c-bd76-373d30e83937 2023-12-02T03:18:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_41 journal status 2023-12-02T03:18:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_41 journal status' 2023-12-02T03:18:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_41 2023-12-02T03:18:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_41 2023-12-02T03:18:21.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:21.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_41 snapshots 2023-12-02T03:18:21.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:21.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_41 snapshots' 2023-12-02T03:18:21.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_41 2023-12-02T03:18:21.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:21.627 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 108 .mirror.non_primary.5db3178e-6670-496c-bd76-373d30e83937.6e047c54-1486-4418-8d94-eb648bc6587a 128 MiB Sat Dec 2 03:07:24 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:108 copied) 2023-12-02T03:18:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:21.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_42 info 2023-12-02T03:18:21.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_42 info' 2023-12-02T03:18:21.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_42 2023-12-02T03:18:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_42': 2023-12-02T03:18:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:18:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12d0dd0c41cb 2023-12-02T03:18:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12d0dd0c41cb 2023-12-02T03:18:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:07:30 2023 2023-12-02T03:18:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:07:30 2023 2023-12-02T03:18:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:07:30 2023 2023-12-02T03:18:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 55cf3916-f72c-47f2-b9d9-b3f8b8496445 2023-12-02T03:18:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_42 journal status 2023-12-02T03:18:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_42 journal status' 2023-12-02T03:18:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_42 2023-12-02T03:18:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_42 2023-12-02T03:18:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_42 snapshots 2023-12-02T03:18:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_42 snapshots' 2023-12-02T03:18:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_42 2023-12-02T03:18:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 109 .mirror.non_primary.55cf3916-f72c-47f2-b9d9-b3f8b8496445.5b3b3a5b-fc84-4b7c-9a40-2aa24f14efce 128 MiB Sat Dec 2 03:07:31 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:109 copied) 2023-12-02T03:18:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_43 info' 2023-12-02T03:18:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_43 info 2023-12-02T03:18:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_43 2023-12-02T03:18:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_43': 2023-12-02T03:18:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:18:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12d0f5967ae8 2023-12-02T03:18:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12d0f5967ae8 2023-12-02T03:18:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:07:43 2023 2023-12-02T03:18:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:07:43 2023 2023-12-02T03:18:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:07:43 2023 2023-12-02T03:18:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 0c9ffe84-d24c-4f74-b087-b033f2cc4422 2023-12-02T03:18:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_43 journal status 2023-12-02T03:18:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_43 journal status' 2023-12-02T03:18:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_43 2023-12-02T03:18:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_43 2023-12-02T03:18:21.856 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_43 snapshots 2023-12-02T03:18:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_43 snapshots' 2023-12-02T03:18:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_43 2023-12-02T03:18:21.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:21.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 110 .mirror.non_primary.0c9ffe84-d24c-4f74-b087-b033f2cc4422.7966bd58-c92f-413b-bcc0-4eff91e2e40b 128 MiB Sat Dec 2 03:07:44 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:110 copied) 2023-12-02T03:18:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:21.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:21.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_44 info 2023-12-02T03:18:21.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_44 info' 2023-12-02T03:18:21.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_44 2023-12-02T03:18:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_44': 2023-12-02T03:18:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:18:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12d024146ee9 2023-12-02T03:18:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12d024146ee9 2023-12-02T03:18:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:07:54 2023 2023-12-02T03:18:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:07:54 2023 2023-12-02T03:18:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:07:54 2023 2023-12-02T03:18:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 2e4177a2-c078-4cd9-b5d2-ee2477a02290 2023-12-02T03:18:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_44 journal status 2023-12-02T03:18:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_44 journal status' 2023-12-02T03:18:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_44 2023-12-02T03:18:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_44 2023-12-02T03:18:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_44 snapshots 2023-12-02T03:18:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_44 snapshots' 2023-12-02T03:18:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_44 2023-12-02T03:18:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 111 .mirror.non_primary.2e4177a2-c078-4cd9-b5d2-ee2477a02290.f6d07b82-b0b8-48a0-90ab-5a8686d0116a 128 MiB Sat Dec 2 03:07:54 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:111 copied) 2023-12-02T03:18:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_45 info' 2023-12-02T03:18:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_45 info 2023-12-02T03:18:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_45 2023-12-02T03:18:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_45': 2023-12-02T03:18:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:18:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 11b81727fad0 2023-12-02T03:18:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.11b81727fad0 2023-12-02T03:18:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:08:01 2023 2023-12-02T03:18:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:08:01 2023 2023-12-02T03:18:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:08:01 2023 2023-12-02T03:18:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: c46da9c3-6d9c-4389-a394-a73d56fb3cdf 2023-12-02T03:18:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:22.082 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_45 journal status 2023-12-02T03:18:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_45 journal status' 2023-12-02T03:18:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_45 2023-12-02T03:18:22.124 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_45 2023-12-02T03:18:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_45 snapshots 2023-12-02T03:18:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_45 snapshots' 2023-12-02T03:18:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_45 2023-12-02T03:18:22.171 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:22.171 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 112 .mirror.non_primary.c46da9c3-6d9c-4389-a394-a73d56fb3cdf.3f3c488a-6b7d-4543-8bdb-a3971f426a17 128 MiB Sat Dec 2 03:08:01 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:112 copied) 2023-12-02T03:18:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_46 info 2023-12-02T03:18:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_46 info' 2023-12-02T03:18:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_46 2023-12-02T03:18:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_46': 2023-12-02T03:18:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:18:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12a165a6d6e 2023-12-02T03:18:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12a165a6d6e 2023-12-02T03:18:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:08:12 2023 2023-12-02T03:18:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:08:12 2023 2023-12-02T03:18:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:08:12 2023 2023-12-02T03:18:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 0a3b7484-3d9a-411e-9ed7-b9ca249bd93a 2023-12-02T03:18:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_46 journal status 2023-12-02T03:18:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_46 journal status' 2023-12-02T03:18:22.220 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_46 2023-12-02T03:18:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_46 2023-12-02T03:18:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_46 snapshots 2023-12-02T03:18:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_46 snapshots' 2023-12-02T03:18:22.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_46 2023-12-02T03:18:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 113 .mirror.non_primary.0a3b7484-3d9a-411e-9ed7-b9ca249bd93a.73cd66c2-dcc5-4484-a88f-29e3edec45fd 128 MiB Sat Dec 2 03:08:13 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:113 copied) 2023-12-02T03:18:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_47 info 2023-12-02T03:18:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_47 info' 2023-12-02T03:18:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_47 2023-12-02T03:18:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_47': 2023-12-02T03:18:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:18:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12b167d43ea3 2023-12-02T03:18:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12b167d43ea3 2023-12-02T03:18:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:08:26 2023 2023-12-02T03:18:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:08:26 2023 2023-12-02T03:18:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:08:26 2023 2023-12-02T03:18:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: be8f73e3-45b7-4b7e-8479-7fd35126d17d 2023-12-02T03:18:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_47 journal status 2023-12-02T03:18:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_47 journal status' 2023-12-02T03:18:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_47 2023-12-02T03:18:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_47 2023-12-02T03:18:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_47 snapshots 2023-12-02T03:18:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_47 snapshots' 2023-12-02T03:18:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_47 2023-12-02T03:18:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 114 .mirror.non_primary.be8f73e3-45b7-4b7e-8479-7fd35126d17d.bc7c8fa1-9b2a-4b24-90e1-ac06affe916e 128 MiB Sat Dec 2 03:08:26 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:114 copied) 2023-12-02T03:18:22.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:22.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:22.452 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_48 info 2023-12-02T03:18:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_48 info' 2023-12-02T03:18:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_48 2023-12-02T03:18:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_48': 2023-12-02T03:18:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:18:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12d0cc754d64 2023-12-02T03:18:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12d0cc754d64 2023-12-02T03:18:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:08:35 2023 2023-12-02T03:18:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:08:35 2023 2023-12-02T03:18:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:08:35 2023 2023-12-02T03:18:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 4f77376e-57d2-4b09-b376-6129ff7390b9 2023-12-02T03:18:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_48 journal status 2023-12-02T03:18:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_48 journal status' 2023-12-02T03:18:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_48 2023-12-02T03:18:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_48 2023-12-02T03:18:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_48 snapshots 2023-12-02T03:18:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_48 snapshots' 2023-12-02T03:18:22.543 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_48 2023-12-02T03:18:22.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:22.584 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 115 .mirror.non_primary.4f77376e-57d2-4b09-b376-6129ff7390b9.c0627aee-1c62-4bfe-9f00-8d52b1a46e87 128 MiB Sat Dec 2 03:08:35 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:115 copied) 2023-12-02T03:18:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_49 info' 2023-12-02T03:18:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_49 info 2023-12-02T03:18:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_49 2023-12-02T03:18:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_49': 2023-12-02T03:18:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:18:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 11b88475b126 2023-12-02T03:18:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.11b88475b126 2023-12-02T03:18:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:08:41 2023 2023-12-02T03:18:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:08:41 2023 2023-12-02T03:18:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:08:41 2023 2023-12-02T03:18:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 92a00385-f1d8-4e86-abb7-0e1810b799c3 2023-12-02T03:18:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_49 journal status 2023-12-02T03:18:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_49 journal status' 2023-12-02T03:18:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_49 2023-12-02T03:18:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_49 2023-12-02T03:18:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_49 snapshots' 2023-12-02T03:18:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_49 snapshots 2023-12-02T03:18:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_49 2023-12-02T03:18:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:22.718 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 116 .mirror.non_primary.92a00385-f1d8-4e86-abb7-0e1810b799c3.fbfa4fa0-a884-4481-ab18-ccbb8233cf30 128 MiB Sat Dec 2 03:08:41 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:116 copied) 2023-12-02T03:18:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_5 info 2023-12-02T03:18:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_5 info' 2023-12-02T03:18:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_5 2023-12-02T03:18:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_5': 2023-12-02T03:18:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:22.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:22.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:18:22.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12a1cb5e484e 2023-12-02T03:18:22.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12a1cb5e484e 2023-12-02T03:18:22.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:22.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:22.762 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:00:27 2023 2023-12-02T03:18:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:10:35 2023 2023-12-02T03:18:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:00:27 2023 2023-12-02T03:18:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 19ecaa0e-982b-435c-a3a7-d881b478cee6 2023-12-02T03:18:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_5 journal status 2023-12-02T03:18:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_5 journal status' 2023-12-02T03:18:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_5 2023-12-02T03:18:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_5 2023-12-02T03:18:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_5 snapshots 2023-12-02T03:18:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_5 snapshots' 2023-12-02T03:18:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_5 2023-12-02T03:18:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 130 snap 128 MiB Sat Dec 2 03:10:32 2023 user 2023-12-02T03:18:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 131 .mirror.non_primary.19ecaa0e-982b-435c-a3a7-d881b478cee6.ce5f0e66-04b5-42ec-bb95-90d4ad5543d5 128 MiB Sat Dec 2 03:10:33 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:127 copied) 2023-12-02T03:18:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_50 info 2023-12-02T03:18:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_50 info' 2023-12-02T03:18:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_50 2023-12-02T03:18:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_50': 2023-12-02T03:18:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 1 2023-12-02T03:18:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12a1534fff16 2023-12-02T03:18:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12a1534fff16 2023-12-02T03:18:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:08:56 2023 2023-12-02T03:18:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:08:56 2023 2023-12-02T03:18:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:08:56 2023 2023-12-02T03:18:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: f939d359-e3e3-4881-b4c7-c05064e62969 2023-12-02T03:18:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:22.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:22.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_50 journal status 2023-12-02T03:18:22.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:22.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_50 journal status' 2023-12-02T03:18:22.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_50 2023-12-02T03:18:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_50 2023-12-02T03:18:22.941 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:22.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:22.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_50 snapshots 2023-12-02T03:18:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_50 snapshots' 2023-12-02T03:18:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_50 2023-12-02T03:18:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 117 .mirror.non_primary.f939d359-e3e3-4881-b4c7-c05064e62969.f0472c87-f3a1-4b67-baf2-bd73732c27a2 128 MiB Sat Dec 2 03:08:56 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:117 copied) 2023-12-02T03:18:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_6 info 2023-12-02T03:18:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_6 info' 2023-12-02T03:18:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_6 2023-12-02T03:18:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_6': 2023-12-02T03:18:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:18:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 11b85bdab31 2023-12-02T03:18:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.11b85bdab31 2023-12-02T03:18:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:00:36 2023 2023-12-02T03:18:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:10:41 2023 2023-12-02T03:18:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:00:36 2023 2023-12-02T03:18:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: f1a9b266-fc3d-4246-8a57-fe1695af4b4d 2023-12-02T03:18:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:23.030 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.030 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_6 journal status 2023-12-02T03:18:23.030 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:23.031 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_6 journal status' 2023-12-02T03:18:23.031 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_6 2023-12-02T03:18:23.072 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_6 2023-12-02T03:18:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_6 snapshots 2023-12-02T03:18:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_6 snapshots' 2023-12-02T03:18:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_6 2023-12-02T03:18:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 133 snap 128 MiB Sat Dec 2 03:10:38 2023 user 2023-12-02T03:18:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 134 .mirror.non_primary.f1a9b266-fc3d-4246-8a57-fe1695af4b4d.b4feff95-89c1-43d0-9ccf-0b48a8ad0105 128 MiB Sat Dec 2 03:10:39 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:129 copied) 2023-12-02T03:18:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_7 info 2023-12-02T03:18:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_7 info' 2023-12-02T03:18:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_7 2023-12-02T03:18:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_7': 2023-12-02T03:18:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:18:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 121182e7c028 2023-12-02T03:18:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.121182e7c028 2023-12-02T03:18:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:00:47 2023 2023-12-02T03:18:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:10:47 2023 2023-12-02T03:18:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:00:47 2023 2023-12-02T03:18:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: 14b5094b-8df2-410f-b1dd-ae91594e9848 2023-12-02T03:18:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_7 journal status' 2023-12-02T03:18:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_7 journal status 2023-12-02T03:18:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_7 2023-12-02T03:18:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_7 2023-12-02T03:18:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_7 snapshots 2023-12-02T03:18:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_7 snapshots' 2023-12-02T03:18:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_7 2023-12-02T03:18:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 136 snap 128 MiB Sat Dec 2 03:10:44 2023 user 2023-12-02T03:18:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 137 .mirror.non_primary.14b5094b-8df2-410f-b1dd-ae91594e9848.8f3ae2a9-7840-42ab-80c7-de3bd2a0021a 128 MiB Sat Dec 2 03:10:45 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:131 copied) 2023-12-02T03:18:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_8 info' 2023-12-02T03:18:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_8 info 2023-12-02T03:18:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_8 2023-12-02T03:18:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_8': 2023-12-02T03:18:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:18:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 12a15bf374b1 2023-12-02T03:18:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.12a15bf374b1 2023-12-02T03:18:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:00:58 2023 2023-12-02T03:18:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:10:53 2023 2023-12-02T03:18:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:00:58 2023 2023-12-02T03:18:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: f71e0710-9762-4b0b-8375-4f15e3c521f1 2023-12-02T03:18:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:23.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_8 journal status 2023-12-02T03:18:23.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:23.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_8 journal status' 2023-12-02T03:18:23.303 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_8 2023-12-02T03:18:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_8 2023-12-02T03:18:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_8 snapshots 2023-12-02T03:18:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_8 snapshots' 2023-12-02T03:18:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_8 2023-12-02T03:18:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 139 snap 128 MiB Sat Dec 2 03:10:50 2023 user 2023-12-02T03:18:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 140 .mirror.non_primary.f71e0710-9762-4b0b-8375-4f15e3c521f1.05d2333e-8519-4b25-850c-9a6a02d1ddeb 128 MiB Sat Dec 2 03:10:51 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:133 copied) 2023-12-02T03:18:23.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-02T03:18:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_9 info 2023-12-02T03:18:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_9 info' 2023-12-02T03:18:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_9 2023-12-02T03:18:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:rbd image 'image_9': 2023-12-02T03:18:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: size 128 MiB in 32 objects 2023-12-02T03:18:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: order 22 (4 MiB objects) 2023-12-02T03:18:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: snapshot_count: 2 2023-12-02T03:18:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: id: 10e49b249977 2023-12-02T03:18:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: block_name_prefix: rbd_data.10e49b249977 2023-12-02T03:18:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: format: 2 2023-12-02T03:18:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: features: layering, non-primary 2023-12-02T03:18:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: op_features: 2023-12-02T03:18:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: flags: 2023-12-02T03:18:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: create_timestamp: Sat Dec 2 03:01:13 2023 2023-12-02T03:18:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: access_timestamp: Sat Dec 2 03:10:59 2023 2023-12-02T03:18:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: modify_timestamp: Sat Dec 2 03:01:13 2023 2023-12-02T03:18:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring state: enabled 2023-12-02T03:18:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring mode: snapshot 2023-12-02T03:18:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring global id: da41f19a-67cd-4518-a939-22e8fe4b6971 2023-12-02T03:18:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: mirroring primary: false 2023-12-02T03:18:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_9 journal status 2023-12-02T03:18:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_9 journal status' 2023-12-02T03:18:23.644 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_9 2023-12-02T03:18:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: journaling is not enabled for image image_9 2023-12-02T03:18:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image image_9 snapshots 2023-12-02T03:18:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'image image_9 snapshots' 2023-12-02T03:18:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_9 2023-12-02T03:18:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-02T03:18:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 142 snap 128 MiB Sat Dec 2 03:10:56 2023 user 2023-12-02T03:18:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 143 .mirror.non_primary.da41f19a-67cd-4518-a939-22e8fe4b6971.27a6b50a-9de7-4175-a0b9-611f9463e5da 128 MiB Sat Dec 2 03:10:57 2023 mirror (non-primary peer_uuids:[] 8bf924be-d648-46b9-b43a-ae071352b043:135 copied) 2023-12-02T03:18:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:18:23.738 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.738 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster1 mirror rbd_mirroring omap vals' 2023-12-02T03:18:23.738 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster1 mirror rbd_mirroring omap vals 2023-12-02T03:18:23.738 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rados --cluster cluster1 -p mirror --namespace '' listomapvals rbd_mirroring 2023-12-02T03:18:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_03c2b3e5-9969-4a06-b152-32b7163be7b1 2023-12-02T03:18:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 61 31 34 61 32 61 62 32 38 63 |....12a14a2ab28c| 2023-12-02T03:18:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_09e06fe9-a175-4009-952b-403e63e1ac10 2023-12-02T03:18:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 31 35 65 66 34 64 64 35 63 31 64 |....115ef4dd5c1d| 2023-12-02T03:18:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_0a3b7484-3d9a-411e-9ed7-b9ca249bd93a 2023-12-02T03:18:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (15 bytes) : 2023-12-02T03:18:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0b 00 00 00 31 32 61 31 36 35 61 36 64 36 65 |....12a165a6d6e| 2023-12-02T03:18:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000000f 2023-12-02T03:18:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_0c9ffe84-d24c-4f74-b087-b033f2cc4422 2023-12-02T03:18:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 64 30 66 35 39 36 37 61 65 38 |....12d0f5967ae8| 2023-12-02T03:18:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_14b5094b-8df2-410f-b1dd-ae91594e9848 2023-12-02T03:18:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 31 31 38 32 65 37 63 30 32 38 |....121182e7c028| 2023-12-02T03:18:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_19ecaa0e-982b-435c-a3a7-d881b478cee6 2023-12-02T03:18:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 61 31 63 62 35 65 34 38 34 65 |....12a1cb5e484e| 2023-12-02T03:18:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_1d600ef5-05fa-4ec8-bb36-7d891860097c 2023-12-02T03:18:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (15 bytes) : 2023-12-02T03:18:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0b 00 00 00 31 31 62 38 33 38 38 35 31 62 33 |....11b838851b3| 2023-12-02T03:18:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000000f 2023-12-02T03:18:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_1f4ea1ff-7710-42b2-9726-194a70ed8f1e 2023-12-02T03:18:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (15 bytes) : 2023-12-02T03:18:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0b 00 00 00 31 32 62 31 63 64 36 36 32 33 38 |....12b1cd66238| 2023-12-02T03:18:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000000f 2023-12-02T03:18:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_258c1b03-48a5-42f2-8745-ac0283c8bc87 2023-12-02T03:18:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 31 31 35 65 39 66 37 37 64 37 |....12115e9f77d7| 2023-12-02T03:18:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_26af4e3d-22a4-46d1-af96-3f3836056461 2023-12-02T03:18:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 31 62 38 63 30 65 38 64 66 66 38 |....11b8c0e8dff8| 2023-12-02T03:18:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_2e4177a2-c078-4cd9-b5d2-ee2477a02290 2023-12-02T03:18:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 64 30 32 34 31 34 36 65 65 39 |....12d024146ee9| 2023-12-02T03:18:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_3d00c370-35ca-4837-a707-0aa3e5b4c7f4 2023-12-02T03:18:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 31 62 38 32 65 65 33 33 36 62 62 |....11b82ee336bb| 2023-12-02T03:18:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_453a5e01-39a4-415e-bf21-d59460038eff 2023-12-02T03:18:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 62 31 66 39 64 39 38 62 62 36 |....12b1f9d98bb6| 2023-12-02T03:18:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_49c16484-3472-4671-8be8-03c1846564a3 2023-12-02T03:18:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 30 65 34 63 66 30 66 64 35 62 32 |....10e4cf0fd5b2| 2023-12-02T03:18:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_4f77376e-57d2-4b09-b376-6129ff7390b9 2023-12-02T03:18:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 64 30 63 63 37 35 34 64 36 34 |....12d0cc754d64| 2023-12-02T03:18:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_55cf3916-f72c-47f2-b9d9-b3f8b8496445 2023-12-02T03:18:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 64 30 64 64 30 63 34 31 63 62 |....12d0dd0c41cb| 2023-12-02T03:18:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_56d75e57-f434-4172-81f2-851edf5c5eb5 2023-12-02T03:18:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 31 62 38 32 31 32 30 32 30 38 33 |....11b821202083| 2023-12-02T03:18:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_5a34854e-65bf-43d7-9d2c-e0481cc5a159 2023-12-02T03:18:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 31 31 64 61 37 64 64 37 39 63 |....1211da7dd79c| 2023-12-02T03:18:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_5c374561-6563-424e-bb48-6c5014bde1fc 2023-12-02T03:18:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 63 30 36 37 35 66 30 33 62 32 |....12c0675f03b2| 2023-12-02T03:18:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_5db3178e-6670-496c-bd76-373d30e83937 2023-12-02T03:18:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 31 62 38 62 65 64 34 38 33 37 63 |....11b8bed4837c| 2023-12-02T03:18:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_730af9f3-f6b6-4cf2-9474-5684c31f584d 2023-12-02T03:18:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 31 31 37 35 66 63 35 39 63 35 |....121175fc59c5| 2023-12-02T03:18:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_7dc110f7-a667-4d4c-9e52-4c6fe497aadb 2023-12-02T03:18:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (15 bytes) : 2023-12-02T03:18:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0b 00 00 00 31 32 61 31 66 31 38 31 37 61 63 |....12a1f1817ac| 2023-12-02T03:18:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000000f 2023-12-02T03:18:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_807f0ce9-67ef-4e3a-987e-bfe34f97a832 2023-12-02T03:18:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 62 31 65 31 63 36 33 31 32 61 |....12b1e1c6312a| 2023-12-02T03:18:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_8117a00a-84c1-4b06-96bd-60840c20cce4 2023-12-02T03:18:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 62 31 66 39 36 66 65 32 38 32 |....12b1f96fe282| 2023-12-02T03:18:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_8696705f-855b-402d-afd6-1c4d55e06c91 2023-12-02T03:18:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (15 bytes) : 2023-12-02T03:18:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0b 00 00 00 31 32 63 30 62 30 30 65 30 30 35 |....12c0b00e005| 2023-12-02T03:18:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000000f 2023-12-02T03:18:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_8a609a08-21d0-4193-a27d-f581f4516454 2023-12-02T03:18:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 61 31 32 34 31 34 31 61 39 37 |....12a124141a97| 2023-12-02T03:18:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_8ef4933a-3981-486a-8a90-10311d67bf00 2023-12-02T03:18:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 62 31 65 64 38 65 36 31 37 63 |....12b1ed8e617c| 2023-12-02T03:18:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_92a00385-f1d8-4e86-abb7-0e1810b799c3 2023-12-02T03:18:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 31 62 38 38 34 37 35 62 31 32 36 |....11b88475b126| 2023-12-02T03:18:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_9333a204-121f-424e-849b-e6aec78f4224 2023-12-02T03:18:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 31 62 38 63 32 31 65 63 61 33 37 |....11b8c21eca37| 2023-12-02T03:18:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_9802897e-7ccb-4bb2-8ab0-cedc09b0044e 2023-12-02T03:18:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 62 31 63 31 65 39 62 64 39 65 |....12b1c1e9bd9e| 2023-12-02T03:18:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_9b3405db-a7dc-43f6-8462-f7325ff11d39 2023-12-02T03:18:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 63 30 35 63 39 31 65 66 37 30 |....12c05c91ef70| 2023-12-02T03:18:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_a30b9861-e056-4151-bcfe-e4a4f2e867dd 2023-12-02T03:18:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 61 31 31 30 65 33 64 38 32 31 |....12a110e3d821| 2023-12-02T03:18:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_a35ab54d-17a7-446d-984d-02da30173d31 2023-12-02T03:18:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 31 62 38 63 31 30 36 64 34 64 35 |....11b8c106d4d5| 2023-12-02T03:18:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_b629353c-89f1-46c0-b5ff-f652e93d5cb3 2023-12-02T03:18:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 31 62 38 39 33 36 62 35 37 35 36 |....11b8936b5756| 2023-12-02T03:18:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_b8d1a79d-fa19-4b64-ba6b-5a4658335ba3 2023-12-02T03:18:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 31 31 38 64 35 31 61 38 36 38 |....12118d51a868| 2023-12-02T03:18:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_be8f73e3-45b7-4b7e-8479-7fd35126d17d 2023-12-02T03:18:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 62 31 36 37 64 34 33 65 61 33 |....12b167d43ea3| 2023-12-02T03:18:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_c1ab0b04-68aa-4508-ad3f-25906784bf9e 2023-12-02T03:18:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 62 31 34 31 30 39 66 38 35 63 |....12b14109f85c| 2023-12-02T03:18:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_c46da9c3-6d9c-4389-a394-a73d56fb3cdf 2023-12-02T03:18:23.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 31 62 38 31 37 32 37 66 61 64 30 |....11b81727fad0| 2023-12-02T03:18:23.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_cdf57a21-3ee2-466a-9e94-c4dd4500ddd1 2023-12-02T03:18:23.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 30 65 34 39 65 63 33 39 39 30 39 |....10e49ec39909| 2023-12-02T03:18:23.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_d4ceac3d-c782-416e-a7e1-be445a43ae35 2023-12-02T03:18:23.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 63 30 32 35 33 36 64 39 36 38 |....12c02536d968| 2023-12-02T03:18:23.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.831 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_da41f19a-67cd-4518-a939-22e8fe4b6971 2023-12-02T03:18:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 30 65 34 39 62 32 34 39 39 37 37 |....10e49b249977| 2023-12-02T03:18:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_ddeae318-0d59-4bf2-b55e-a3f9934ab1b6 2023-12-02T03:18:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 31 62 38 62 32 31 64 39 35 62 63 |....11b8b21d95bc| 2023-12-02T03:18:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_e2407e91-d52b-4a70-b3dd-c23362ba73c5 2023-12-02T03:18:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 61 31 32 35 33 38 39 65 32 31 |....12a125389e21| 2023-12-02T03:18:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_e7466612-311c-416f-a7f9-bfa20e74a575 2023-12-02T03:18:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 63 30 64 38 35 38 64 35 33 32 |....12c0d858d532| 2023-12-02T03:18:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_f12eeb37-6efd-4241-9808-5371a9611fae 2023-12-02T03:18:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 63 30 34 31 33 30 35 66 35 64 |....12c041305f5d| 2023-12-02T03:18:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_f1a9b266-fc3d-4246-8a57-fe1695af4b4d 2023-12-02T03:18:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (15 bytes) : 2023-12-02T03:18:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0b 00 00 00 31 31 62 38 35 62 64 61 62 33 31 |....11b85bdab31| 2023-12-02T03:18:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000000f 2023-12-02T03:18:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_f71e0710-9762-4b0b-8375-4f15e3c521f1 2023-12-02T03:18:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 61 31 35 62 66 33 37 34 62 31 |....12a15bf374b1| 2023-12-02T03:18:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_f939d359-e3e3-4881-b4c7-c05064e62969 2023-12-02T03:18:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 61 31 35 33 34 66 66 66 31 36 |....12a1534fff16| 2023-12-02T03:18:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_faedaa7c-e337-491d-80ec-3cfa45a9493c 2023-12-02T03:18:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 61 31 66 36 30 61 34 64 61 64 |....12a1f60a4dad| 2023-12-02T03:18:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:global_ffc4290c-3b4c-45f6-a30f-c93f3cb26275 2023-12-02T03:18:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (16 bytes) : 2023-12-02T03:18:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 0c 00 00 00 31 32 62 31 63 31 66 66 63 62 64 61 |....12b1c1ffcbda| 2023-12-02T03:18:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 2023-12-02T03:18:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_10e49b249977 2023-12-02T03:18:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 64 61 34 31 66 31 |..*...$...da41f1| 2023-12-02T03:18:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 39 61 2d 36 37 63 64 2d 34 35 31 38 2d 61 39 33 |9a-67cd-4518-a93| 2023-12-02T03:18:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 39 2d 32 32 65 38 66 65 34 62 36 39 37 31 01 01 |9-22e8fe4b6971..| 2023-12-02T03:18:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_10e49ec39909 2023-12-02T03:18:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 63 64 66 35 37 61 |..*...$...cdf57a| 2023-12-02T03:18:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 32 31 2d 33 65 65 32 2d 34 36 36 61 2d 39 65 39 |21-3ee2-466a-9e9| 2023-12-02T03:18:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 34 2d 63 34 64 64 34 35 30 30 64 64 64 31 01 01 |4-c4dd4500ddd1..| 2023-12-02T03:18:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_10e4cf0fd5b2 2023-12-02T03:18:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 34 39 63 31 36 34 |..*...$...49c164| 2023-12-02T03:18:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 38 34 2d 33 34 37 32 2d 34 36 37 31 2d 38 62 65 |84-3472-4671-8be| 2023-12-02T03:18:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 38 2d 30 33 63 31 38 34 36 35 36 34 61 33 01 01 |8-03c1846564a3..| 2023-12-02T03:18:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_115ef4dd5c1d 2023-12-02T03:18:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 30 39 65 30 36 66 |..*...$...09e06f| 2023-12-02T03:18:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 65 39 2d 61 31 37 35 2d 34 30 30 39 2d 39 35 32 |e9-a175-4009-952| 2023-12-02T03:18:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 62 2d 34 30 33 65 36 33 65 31 61 63 31 30 01 01 |b-403e63e1ac10..| 2023-12-02T03:18:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_11b81727fad0 2023-12-02T03:18:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 63 34 36 64 61 39 |..*...$...c46da9| 2023-12-02T03:18:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 63 33 2d 36 64 39 63 2d 34 33 38 39 2d 61 33 39 |c3-6d9c-4389-a39| 2023-12-02T03:18:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 34 2d 61 37 33 64 35 36 66 62 33 63 64 66 01 01 |4-a73d56fb3cdf..| 2023-12-02T03:18:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_11b821202083 2023-12-02T03:18:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 35 36 64 37 35 65 |..*...$...56d75e| 2023-12-02T03:18:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 35 37 2d 66 34 33 34 2d 34 31 37 32 2d 38 31 66 |57-f434-4172-81f| 2023-12-02T03:18:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 32 2d 38 35 31 65 64 66 35 63 35 65 62 35 01 01 |2-851edf5c5eb5..| 2023-12-02T03:18:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_11b82ee336bb 2023-12-02T03:18:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 33 64 30 30 63 33 |..*...$...3d00c3| 2023-12-02T03:18:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 37 30 2d 33 35 63 61 2d 34 38 33 37 2d 61 37 30 |70-35ca-4837-a70| 2023-12-02T03:18:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 37 2d 30 61 61 33 65 35 62 34 63 37 66 34 01 01 |7-0aa3e5b4c7f4..| 2023-12-02T03:18:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_11b838851b3 2023-12-02T03:18:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 31 64 36 30 30 65 |..*...$...1d600e| 2023-12-02T03:18:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 66 35 2d 30 35 66 61 2d 34 65 63 38 2d 62 62 33 |f5-05fa-4ec8-bb3| 2023-12-02T03:18:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 36 2d 37 64 38 39 31 38 36 30 30 39 37 63 01 01 |6-7d891860097c..| 2023-12-02T03:18:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_11b85bdab31 2023-12-02T03:18:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 66 31 61 39 62 32 |..*...$...f1a9b2| 2023-12-02T03:18:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 36 36 2d 66 63 33 64 2d 34 32 34 36 2d 38 61 35 |66-fc3d-4246-8a5| 2023-12-02T03:18:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 37 2d 66 65 31 36 39 35 61 66 34 62 34 64 01 01 |7-fe1695af4b4d..| 2023-12-02T03:18:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_11b88475b126 2023-12-02T03:18:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 39 32 61 30 30 33 |..*...$...92a003| 2023-12-02T03:18:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 38 35 2d 66 31 64 38 2d 34 65 38 36 2d 61 62 62 |85-f1d8-4e86-abb| 2023-12-02T03:18:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 37 2d 30 65 31 38 31 30 62 37 39 39 63 33 01 01 |7-0e1810b799c3..| 2023-12-02T03:18:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_11b8936b5756 2023-12-02T03:18:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 62 36 32 39 33 35 |..*...$...b62935| 2023-12-02T03:18:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 33 63 2d 38 39 66 31 2d 34 36 63 30 2d 62 35 66 |3c-89f1-46c0-b5f| 2023-12-02T03:18:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 66 2d 66 36 35 32 65 39 33 64 35 63 62 33 01 01 |f-f652e93d5cb3..| 2023-12-02T03:18:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_11b8b21d95bc 2023-12-02T03:18:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 64 64 65 61 65 33 |..*...$...ddeae3| 2023-12-02T03:18:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 31 38 2d 30 64 35 39 2d 34 62 66 32 2d 62 35 35 |18-0d59-4bf2-b55| 2023-12-02T03:18:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 65 2d 61 33 66 39 39 33 34 61 62 31 62 36 01 01 |e-a3f9934ab1b6..| 2023-12-02T03:18:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_11b8bed4837c 2023-12-02T03:18:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 35 64 62 33 31 37 |..*...$...5db317| 2023-12-02T03:18:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 38 65 2d 36 36 37 30 2d 34 39 36 63 2d 62 64 37 |8e-6670-496c-bd7| 2023-12-02T03:18:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 36 2d 33 37 33 64 33 30 65 38 33 39 33 37 01 01 |6-373d30e83937..| 2023-12-02T03:18:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_11b8c0e8dff8 2023-12-02T03:18:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 32 36 61 66 34 65 |..*...$...26af4e| 2023-12-02T03:18:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 33 64 2d 32 32 61 34 2d 34 36 64 31 2d 61 66 39 |3d-22a4-46d1-af9| 2023-12-02T03:18:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 36 2d 33 66 33 38 33 36 30 35 36 34 36 31 01 01 |6-3f3836056461..| 2023-12-02T03:18:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_11b8c106d4d5 2023-12-02T03:18:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 61 33 35 61 62 35 |..*...$...a35ab5| 2023-12-02T03:18:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 34 64 2d 31 37 61 37 2d 34 34 36 64 2d 39 38 34 |4d-17a7-446d-984| 2023-12-02T03:18:23.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 64 2d 30 32 64 61 33 30 31 37 33 64 33 31 01 01 |d-02da30173d31..| 2023-12-02T03:18:23.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_11b8c21eca37 2023-12-02T03:18:23.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 39 33 33 33 61 32 |..*...$...9333a2| 2023-12-02T03:18:23.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 30 34 2d 31 32 31 66 2d 34 32 34 65 2d 38 34 39 |04-121f-424e-849| 2023-12-02T03:18:23.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 62 2d 65 36 61 65 63 37 38 66 34 32 32 34 01 01 |b-e6aec78f4224..| 2023-12-02T03:18:23.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12115e9f77d7 2023-12-02T03:18:23.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 32 35 38 63 31 62 |..*...$...258c1b| 2023-12-02T03:18:23.843 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 30 33 2d 34 38 61 35 2d 34 32 66 32 2d 38 37 34 |03-48a5-42f2-874| 2023-12-02T03:18:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 35 2d 61 63 30 32 38 33 63 38 62 63 38 37 01 01 |5-ac0283c8bc87..| 2023-12-02T03:18:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_121175fc59c5 2023-12-02T03:18:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 37 33 30 61 66 39 |..*...$...730af9| 2023-12-02T03:18:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 66 33 2d 66 36 62 36 2d 34 63 66 32 2d 39 34 37 |f3-f6b6-4cf2-947| 2023-12-02T03:18:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 34 2d 35 36 38 34 63 33 31 66 35 38 34 64 01 01 |4-5684c31f584d..| 2023-12-02T03:18:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_121182e7c028 2023-12-02T03:18:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 31 34 62 35 30 39 |..*...$...14b509| 2023-12-02T03:18:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 34 62 2d 38 64 66 32 2d 34 31 30 66 2d 62 31 64 |4b-8df2-410f-b1d| 2023-12-02T03:18:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 64 2d 61 65 39 31 35 39 34 65 39 38 34 38 01 01 |d-ae91594e9848..| 2023-12-02T03:18:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12118d51a868 2023-12-02T03:18:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 62 38 64 31 61 37 |..*...$...b8d1a7| 2023-12-02T03:18:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 39 64 2d 66 61 31 39 2d 34 62 36 34 2d 62 61 36 |9d-fa19-4b64-ba6| 2023-12-02T03:18:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 62 2d 35 61 34 36 35 38 33 33 35 62 61 33 01 01 |b-5a4658335ba3..| 2023-12-02T03:18:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_1211da7dd79c 2023-12-02T03:18:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 35 61 33 34 38 35 |..*...$...5a3485| 2023-12-02T03:18:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 34 65 2d 36 35 62 66 2d 34 33 64 37 2d 39 64 32 |4e-65bf-43d7-9d2| 2023-12-02T03:18:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 63 2d 65 30 34 38 31 63 63 35 61 31 35 39 01 01 |c-e0481cc5a159..| 2023-12-02T03:18:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12a110e3d821 2023-12-02T03:18:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 61 33 30 62 39 38 |..*...$...a30b98| 2023-12-02T03:18:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 36 31 2d 65 30 35 36 2d 34 31 35 31 2d 62 63 66 |61-e056-4151-bcf| 2023-12-02T03:18:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 65 2d 65 34 61 34 66 32 65 38 36 37 64 64 01 01 |e-e4a4f2e867dd..| 2023-12-02T03:18:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12a124141a97 2023-12-02T03:18:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 38 61 36 30 39 61 |..*...$...8a609a| 2023-12-02T03:18:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 30 38 2d 32 31 64 30 2d 34 31 39 33 2d 61 32 37 |08-21d0-4193-a27| 2023-12-02T03:18:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 64 2d 66 35 38 31 66 34 35 31 36 34 35 34 01 01 |d-f581f4516454..| 2023-12-02T03:18:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12a125389e21 2023-12-02T03:18:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 65 32 34 30 37 65 |..*...$...e2407e| 2023-12-02T03:18:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 39 31 2d 64 35 32 62 2d 34 61 37 30 2d 62 33 64 |91-d52b-4a70-b3d| 2023-12-02T03:18:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 64 2d 63 32 33 33 36 32 62 61 37 33 63 35 01 01 |d-c23362ba73c5..| 2023-12-02T03:18:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12a14a2ab28c 2023-12-02T03:18:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 30 33 63 32 62 33 |..*...$...03c2b3| 2023-12-02T03:18:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 65 35 2d 39 39 36 39 2d 34 61 30 36 2d 62 31 35 |e5-9969-4a06-b15| 2023-12-02T03:18:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 32 2d 33 32 62 37 31 36 33 62 65 37 62 31 01 01 |2-32b7163be7b1..| 2023-12-02T03:18:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12a1534fff16 2023-12-02T03:18:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 66 39 33 39 64 33 |..*...$...f939d3| 2023-12-02T03:18:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 35 39 2d 65 33 65 33 2d 34 38 38 31 2d 62 34 63 |59-e3e3-4881-b4c| 2023-12-02T03:18:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 37 2d 63 30 35 30 36 34 65 36 32 39 36 39 01 01 |7-c05064e62969..| 2023-12-02T03:18:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12a15bf374b1 2023-12-02T03:18:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 66 37 31 65 30 37 |..*...$...f71e07| 2023-12-02T03:18:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 31 30 2d 39 37 36 32 2d 34 62 30 62 2d 38 33 37 |10-9762-4b0b-837| 2023-12-02T03:18:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 35 2d 34 66 31 35 65 33 63 35 32 31 66 31 01 01 |5-4f15e3c521f1..| 2023-12-02T03:18:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12a165a6d6e 2023-12-02T03:18:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 30 61 33 62 37 34 |..*...$...0a3b74| 2023-12-02T03:18:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 38 34 2d 33 64 39 61 2d 34 31 31 65 2d 39 65 64 |84-3d9a-411e-9ed| 2023-12-02T03:18:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 37 2d 62 39 63 61 32 34 39 62 64 39 33 61 01 01 |7-b9ca249bd93a..| 2023-12-02T03:18:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12a1cb5e484e 2023-12-02T03:18:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 31 39 65 63 61 61 |..*...$...19ecaa| 2023-12-02T03:18:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 30 65 2d 39 38 32 62 2d 34 33 35 63 2d 61 33 61 |0e-982b-435c-a3a| 2023-12-02T03:18:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 37 2d 64 38 38 31 62 34 37 38 63 65 65 36 01 01 |7-d881b478cee6..| 2023-12-02T03:18:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12a1f1817ac 2023-12-02T03:18:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 37 64 63 31 31 30 |..*...$...7dc110| 2023-12-02T03:18:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 66 37 2d 61 36 36 37 2d 34 64 34 63 2d 39 65 35 |f7-a667-4d4c-9e5| 2023-12-02T03:18:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 32 2d 34 63 36 66 65 34 39 37 61 61 64 62 01 01 |2-4c6fe497aadb..| 2023-12-02T03:18:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12a1f60a4dad 2023-12-02T03:18:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 66 61 65 64 61 61 |..*...$...faedaa| 2023-12-02T03:18:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 37 63 2d 65 33 33 37 2d 34 39 31 64 2d 38 30 65 |7c-e337-491d-80e| 2023-12-02T03:18:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 63 2d 33 63 66 61 34 35 61 39 34 39 33 63 01 01 |c-3cfa45a9493c..| 2023-12-02T03:18:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12b14109f85c 2023-12-02T03:18:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 63 31 61 62 30 62 |..*...$...c1ab0b| 2023-12-02T03:18:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 30 34 2d 36 38 61 61 2d 34 35 30 38 2d 61 64 33 |04-68aa-4508-ad3| 2023-12-02T03:18:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 66 2d 32 35 39 30 36 37 38 34 62 66 39 65 01 01 |f-25906784bf9e..| 2023-12-02T03:18:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12b167d43ea3 2023-12-02T03:18:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 62 65 38 66 37 33 |..*...$...be8f73| 2023-12-02T03:18:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 65 33 2d 34 35 62 37 2d 34 62 37 65 2d 38 34 37 |e3-45b7-4b7e-847| 2023-12-02T03:18:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 39 2d 37 66 64 33 35 31 32 36 64 31 37 64 01 01 |9-7fd35126d17d..| 2023-12-02T03:18:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12b1c1e9bd9e 2023-12-02T03:18:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 39 38 30 32 38 39 |..*...$...980289| 2023-12-02T03:18:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 37 65 2d 37 63 63 62 2d 34 62 62 32 2d 38 61 62 |7e-7ccb-4bb2-8ab| 2023-12-02T03:18:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 30 2d 63 65 64 63 30 39 62 30 30 34 34 65 01 01 |0-cedc09b0044e..| 2023-12-02T03:18:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12b1c1ffcbda 2023-12-02T03:18:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 66 66 63 34 32 39 |..*...$...ffc429| 2023-12-02T03:18:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 30 63 2d 33 62 34 63 2d 34 35 66 36 2d 61 33 30 |0c-3b4c-45f6-a30| 2023-12-02T03:18:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 66 2d 63 39 33 66 33 63 62 32 36 32 37 35 01 01 |f-c93f3cb26275..| 2023-12-02T03:18:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12b1cd66238 2023-12-02T03:18:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 31 66 34 65 61 31 |..*...$...1f4ea1| 2023-12-02T03:18:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 66 66 2d 37 37 31 30 2d 34 32 62 32 2d 39 37 32 |ff-7710-42b2-972| 2023-12-02T03:18:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 36 2d 31 39 34 61 37 30 65 64 38 66 31 65 01 01 |6-194a70ed8f1e..| 2023-12-02T03:18:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12b1e1c6312a 2023-12-02T03:18:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 38 30 37 66 30 63 |..*...$...807f0c| 2023-12-02T03:18:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 65 39 2d 36 37 65 66 2d 34 65 33 61 2d 39 38 37 |e9-67ef-4e3a-987| 2023-12-02T03:18:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 65 2d 62 66 65 33 34 66 39 37 61 38 33 32 01 01 |e-bfe34f97a832..| 2023-12-02T03:18:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12b1ed8e617c 2023-12-02T03:18:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 38 65 66 34 39 33 |..*...$...8ef493| 2023-12-02T03:18:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 33 61 2d 33 39 38 31 2d 34 38 36 61 2d 38 61 39 |3a-3981-486a-8a9| 2023-12-02T03:18:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 30 2d 31 30 33 31 31 64 36 37 62 66 30 30 01 01 |0-10311d67bf00..| 2023-12-02T03:18:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12b1f96fe282 2023-12-02T03:18:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 38 31 31 37 61 30 |..*...$...8117a0| 2023-12-02T03:18:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 30 61 2d 38 34 63 31 2d 34 62 30 36 2d 39 36 62 |0a-84c1-4b06-96b| 2023-12-02T03:18:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 64 2d 36 30 38 34 30 63 32 30 63 63 65 34 01 01 |d-60840c20cce4..| 2023-12-02T03:18:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12b1f9d98bb6 2023-12-02T03:18:23.855 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.855 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 34 35 33 61 35 65 |..*...$...453a5e| 2023-12-02T03:18:23.855 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 30 31 2d 33 39 61 34 2d 34 31 35 65 2d 62 66 32 |01-39a4-415e-bf2| 2023-12-02T03:18:23.855 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 31 2d 64 35 39 34 36 30 30 33 38 65 66 66 01 01 |1-d59460038eff..| 2023-12-02T03:18:23.855 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.855 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.855 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12c02536d968 2023-12-02T03:18:23.855 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.855 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 64 34 63 65 61 63 |..*...$...d4ceac| 2023-12-02T03:18:23.855 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 33 64 2d 63 37 38 32 2d 34 31 36 65 2d 61 37 65 |3d-c782-416e-a7e| 2023-12-02T03:18:23.855 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 31 2d 62 65 34 34 35 61 34 33 61 65 33 35 01 01 |1-be445a43ae35..| 2023-12-02T03:18:23.855 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.855 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12c041305f5d 2023-12-02T03:18:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 66 31 32 65 65 62 |..*...$...f12eeb| 2023-12-02T03:18:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 33 37 2d 36 65 66 64 2d 34 32 34 31 2d 39 38 30 |37-6efd-4241-980| 2023-12-02T03:18:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 38 2d 35 33 37 31 61 39 36 31 31 66 61 65 01 01 |8-5371a9611fae..| 2023-12-02T03:18:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12c05c91ef70 2023-12-02T03:18:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 39 62 33 34 30 35 |..*...$...9b3405| 2023-12-02T03:18:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 64 62 2d 61 37 64 63 2d 34 33 66 36 2d 38 34 36 |db-a7dc-43f6-846| 2023-12-02T03:18:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 32 2d 66 37 33 32 35 66 66 31 31 64 33 39 01 01 |2-f7325ff11d39..| 2023-12-02T03:18:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12c0675f03b2 2023-12-02T03:18:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 35 63 33 37 34 35 |..*...$...5c3745| 2023-12-02T03:18:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 36 31 2d 36 35 36 33 2d 34 32 34 65 2d 62 62 34 |61-6563-424e-bb4| 2023-12-02T03:18:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 38 2d 36 63 35 30 31 34 62 64 65 31 66 63 01 01 |8-6c5014bde1fc..| 2023-12-02T03:18:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12c0b00e005 2023-12-02T03:18:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 38 36 39 36 37 30 |..*...$...869670| 2023-12-02T03:18:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 35 66 2d 38 35 35 62 2d 34 30 32 64 2d 61 66 64 |5f-855b-402d-afd| 2023-12-02T03:18:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 36 2d 31 63 34 64 35 35 65 30 36 63 39 31 01 01 |6-1c4d55e06c91..| 2023-12-02T03:18:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12c0d858d532 2023-12-02T03:18:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 65 37 34 36 36 36 |..*...$...e74666| 2023-12-02T03:18:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 31 32 2d 33 31 31 63 2d 34 31 36 66 2d 61 37 66 |12-311c-416f-a7f| 2023-12-02T03:18:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 39 2d 62 66 61 32 30 65 37 34 61 35 37 35 01 01 |9-bfa20e74a575..| 2023-12-02T03:18:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12d024146ee9 2023-12-02T03:18:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 32 65 34 31 37 37 |..*...$...2e4177| 2023-12-02T03:18:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 61 32 2d 63 30 37 38 2d 34 63 64 39 2d 62 35 64 |a2-c078-4cd9-b5d| 2023-12-02T03:18:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 32 2d 65 65 32 34 37 37 61 30 32 32 39 30 01 01 |2-ee2477a02290..| 2023-12-02T03:18:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12d0cc754d64 2023-12-02T03:18:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 34 66 37 37 33 37 |..*...$...4f7737| 2023-12-02T03:18:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 36 65 2d 35 37 64 32 2d 34 62 30 39 2d 62 33 37 |6e-57d2-4b09-b37| 2023-12-02T03:18:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 36 2d 36 31 32 39 66 66 37 33 39 30 62 39 01 01 |6-6129ff7390b9..| 2023-12-02T03:18:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12d0dd0c41cb 2023-12-02T03:18:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 35 35 63 66 33 39 |..*...$...55cf39| 2023-12-02T03:18:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 31 36 2d 66 37 32 63 2d 34 37 66 32 2d 62 39 64 |16-f72c-47f2-b9d| 2023-12-02T03:18:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 39 2d 62 33 66 38 62 38 34 39 36 34 34 35 01 01 |9-b3f8b8496445..| 2023-12-02T03:18:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_12d0f5967ae8 2023-12-02T03:18:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (48 bytes) : 2023-12-02T03:18:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 30 63 39 66 66 65 |..*...$...0c9ffe| 2023-12-02T03:18:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 38 34 2d 64 32 34 63 2d 34 66 37 34 2d 62 30 38 |84-d24c-4f74-b08| 2023-12-02T03:18:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 37 2d 62 30 33 33 66 32 63 63 34 34 32 32 01 01 |7-b033f2cc4422..| 2023-12-02T03:18:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 2023-12-02T03:18:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:mirror_mode 2023-12-02T03:18:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (4 bytes) : 2023-12-02T03:18:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 00 00 00 |....| 2023-12-02T03:18:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000004 2023-12-02T03:18:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:mirror_peer_06fbe0df-d787-44c2-aede-d6f6ace95f83 2023-12-02T03:18:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (132 bytes) : 2023-12-02T03:18:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 02 01 7e 00 00 00 24 00 00 00 30 36 66 62 65 30 |..~...$...06fbe0| 2023-12-02T03:18:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 64 66 2d 64 37 38 37 2d 34 34 63 32 2d 61 65 64 |df-d787-44c2-aed| 2023-12-02T03:18:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 65 2d 64 36 66 36 61 63 65 39 35 66 38 33 08 00 |e-d6f6ace95f83..| 2023-12-02T03:18:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 63 6c 75 73 74 65 72 32 0d 00 00 00 63 6c |..cluster2....cl| 2023-12-02T03:18:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 69 65 6e 74 2e 6d 69 72 72 6f 72 ff ff ff ff ff |ient.mirror.....| 2023-12-02T03:18:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 ff ff ff 02 24 00 00 00 38 62 66 39 32 34 62 65 |....$...8bf924be| 2023-12-02T03:18:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 2d 64 36 34 38 2d 34 36 62 39 2d 62 34 33 61 2d |-d648-46b9-b43a-| 2023-12-02T03:18:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 61 65 30 37 31 33 35 32 62 30 34 33 fa a1 6a 65 |ae071352b043..je| 2023-12-02T03:18:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 0b 33 b4 1a |.3..| 2023-12-02T03:18:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000084 2023-12-02T03:18:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:mirror_uuid 2023-12-02T03:18:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 39 39 37 63 38 30 39 61 2d 62 39 34 35 2d 34 39 |997c809a-b945-49| 2023-12-02T03:18:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 61 62 2d 62 65 33 32 2d 35 63 65 39 64 32 37 34 |ab-be32-5ce9d274| 2023-12-02T03:18:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 32 38 62 65 |28be| 2023-12-02T03:18:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_03c2b3e5-9969-4a06-b152-32b7163be7b1_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 e5 13 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 93 cb e2 98 10 00 |................| 2023-12-02T03:18:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 f8 a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 a9 56 98 12 00 |.V...| 2023-12-02T03:18:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_09e06fe9-a175-4009-952b-403e63e1ac10_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (92 bytes) : 2023-12-02T03:18:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 1d 13 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fc 9f cc 44 10 00 |.............D..| 2023-12-02T03:18:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 24 00 00 00 06 16 00 00 00 6c 6f 63 |....$........loc| 2023-12-02T03:18:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 61 6c 20 69 6d 61 67 65 20 69 73 20 70 72 69 6d |al image is prim| 2023-12-02T03:18:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 61 72 79 b2 a0 6a 65 44 6c b8 12 00 |ary..jeDl...| 2023-12-02T03:18:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000005c 2023-12-02T03:18:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_0a3b7484-3d9a-411e-9ed7-b9ca249bd93a_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 e5 13 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 93 cb e2 98 10 00 |................| 2023-12-02T03:18:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 f8 a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 79 4a 99 12 00 |yJ...| 2023-12-02T03:18:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_0c9ffe84-d24c-4f74-b087-b033f2cc4422_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b7 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 f7 70 f7 60 10 00 |...........p.`..| 2023-12-02T03:18:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 e4 a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 35 f5 a2 17 00 |5....| 2023-12-02T03:18:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_14b5094b-8df2-410f-b1dd-ae91594e9848_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 e5 13 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 93 cb e2 98 10 00 |................| 2023-12-02T03:18:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 f8 a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 01 ef 99 12 00 |.....| 2023-12-02T03:18:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_19ecaa0e-982b-435c-a3a7-d881b478cee6_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (92 bytes) : 2023-12-02T03:18:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 1d 13 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fc 9f cc 44 10 00 |.............D..| 2023-12-02T03:18:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 24 00 00 00 06 16 00 00 00 6c 6f 63 |....$........loc| 2023-12-02T03:18:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 61 6c 20 69 6d 61 67 65 20 69 73 20 70 72 69 6d |al image is prim| 2023-12-02T03:18:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 61 72 79 b2 a0 6a 65 b3 bf f2 12 00 |ary..je.....| 2023-12-02T03:18:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000005c 2023-12-02T03:18:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_1d600ef5-05fa-4ec8-bb36-7d891860097c_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 e5 13 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 93 cb e2 98 10 00 |................| 2023-12-02T03:18:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 f8 a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 6e 87 9a 12 00 |n....| 2023-12-02T03:18:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_1f4ea1ff-7710-42b2-9726-194a70ed8f1e_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (92 bytes) : 2023-12-02T03:18:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 1d 13 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fc 9f cc 44 10 00 |.............D..| 2023-12-02T03:18:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 24 00 00 00 06 16 00 00 00 6c 6f 63 |....$........loc| 2023-12-02T03:18:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 61 6c 20 69 6d 61 67 65 20 69 73 20 70 72 69 6d |al image is prim| 2023-12-02T03:18:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 61 72 79 b2 a0 6a 65 06 36 f4 12 00 |ary..je.6...| 2023-12-02T03:18:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000005c 2023-12-02T03:18:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_258c1b03-48a5-42f2-8745-ac0283c8bc87_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 e5 13 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 93 cb e2 98 10 00 |................| 2023-12-02T03:18:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 f8 a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 07 25 9b 12 00 |.%...| 2023-12-02T03:18:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_26af4e3d-22a4-46d1-af96-3f3836056461_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (92 bytes) : 2023-12-02T03:18:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 1d 13 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fc 9f cc 44 10 00 |.............D..| 2023-12-02T03:18:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 24 00 00 00 06 16 00 00 00 6c 6f 63 |....$........loc| 2023-12-02T03:18:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 61 6c 20 69 6d 61 67 65 20 69 73 20 70 72 69 6d |al image is prim| 2023-12-02T03:18:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 61 72 79 b2 a0 6a 65 9c 99 f5 12 00 |ary..je.....| 2023-12-02T03:18:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000005c 2023-12-02T03:18:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_2e4177a2-c078-4cd9-b5d2-ee2477a02290_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 c6 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 31 74 d0 10 10 00 |..........1t....| 2023-12-02T03:18:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 fa a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 54 b3 4d 15 00 |T.M..| 2023-12-02T03:18:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_3d00c370-35ca-4837-a707-0aa3e5b4c7f4_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 c6 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 31 74 d0 10 10 00 |..........1t....| 2023-12-02T03:18:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 fa a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 37 35 4f 15 00 |75O..| 2023-12-02T03:18:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_453a5e01-39a4-415e-bf21-d59460038eff_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 c6 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 31 74 d0 10 10 00 |..........1t....| 2023-12-02T03:18:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 fa a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 9b 87 50 15 00 |..P..| 2023-12-02T03:18:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_49c16484-3472-4671-8be8-03c1846564a3_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 e5 13 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 93 cb e2 98 10 00 |................| 2023-12-02T03:18:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 f8 a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 fb c3 9b 12 00 |.....| 2023-12-02T03:18:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_4f77376e-57d2-4b09-b376-6129ff7390b9_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 c6 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 31 74 d0 10 10 00 |..........1t....| 2023-12-02T03:18:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 fa a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 a5 bc 51 15 00 |..Q..| 2023-12-02T03:18:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_55cf3916-f72c-47f2-b9d9-b3f8b8496445_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (92 bytes) : 2023-12-02T03:18:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 1d 13 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fc 9f cc 44 10 00 |.............D..| 2023-12-02T03:18:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 24 00 00 00 06 16 00 00 00 6c 6f 63 |....$........loc| 2023-12-02T03:18:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 61 6c 20 69 6d 61 67 65 20 69 73 20 70 72 69 6d |al image is prim| 2023-12-02T03:18:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 61 72 79 b2 a0 6a 65 f9 23 f7 12 00 |ary..je.#...| 2023-12-02T03:18:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000005c 2023-12-02T03:18:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_56d75e57-f434-4172-81f2-851edf5c5eb5_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (92 bytes) : 2023-12-02T03:18:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 1d 13 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fc 9f cc 44 10 00 |.............D..| 2023-12-02T03:18:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 24 00 00 00 06 16 00 00 00 6c 6f 63 |....$........loc| 2023-12-02T03:18:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 61 6c 20 69 6d 61 67 65 20 69 73 20 70 72 69 6d |al image is prim| 2023-12-02T03:18:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 61 72 79 b2 a0 6a 65 85 bd f8 12 00 |ary..je.....| 2023-12-02T03:18:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000005c 2023-12-02T03:18:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_5a34854e-65bf-43d7-9d2c-e0481cc5a159_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 e5 13 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 93 cb e2 98 10 00 |................| 2023-12-02T03:18:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 f8 a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 d7 5c 9c 12 00 |.\...| 2023-12-02T03:18:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_5c374561-6563-424e-bb48-6c5014bde1fc_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 c6 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 31 74 d0 10 10 00 |..........1t....| 2023-12-02T03:18:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 fa a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 f0 ea 52 15 00 |..R..| 2023-12-02T03:18:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_5db3178e-6670-496c-bd76-373d30e83937_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (92 bytes) : 2023-12-02T03:18:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 1d 13 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fc 9f cc 44 10 00 |.............D..| 2023-12-02T03:18:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 24 00 00 00 06 16 00 00 00 6c 6f 63 |....$........loc| 2023-12-02T03:18:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 61 6c 20 69 6d 61 67 65 20 69 73 20 70 72 69 6d |al image is prim| 2023-12-02T03:18:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 61 72 79 b2 a0 6a 65 f6 09 3e 13 00 |ary..je..>..| 2023-12-02T03:18:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000005c 2023-12-02T03:18:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_730af9f3-f6b6-4cf2-9474-5684c31f584d_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (92 bytes) : 2023-12-02T03:18:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 1d 13 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fc 9f cc 44 10 00 |.............D..| 2023-12-02T03:18:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 24 00 00 00 06 16 00 00 00 6c 6f 63 |....$........loc| 2023-12-02T03:18:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 61 6c 20 69 6d 61 67 65 20 69 73 20 70 72 69 6d |al image is prim| 2023-12-02T03:18:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 61 72 79 b2 a0 6a 65 bc 89 3f 13 00 |ary..je..?..| 2023-12-02T03:18:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000005c 2023-12-02T03:18:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_7dc110f7-a667-4d4c-9e52-4c6fe497aadb_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b7 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 f7 70 f7 60 10 00 |...........p.`..| 2023-12-02T03:18:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 e4 a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 12 5a a4 17 00 |.Z...| 2023-12-02T03:18:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_807f0ce9-67ef-4e3a-987e-bfe34f97a832_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 c6 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 31 74 d0 10 10 00 |..........1t....| 2023-12-02T03:18:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 fa a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 32 2a 54 15 00 |2*T..| 2023-12-02T03:18:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_8117a00a-84c1-4b06-96bd-60840c20cce4_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 c6 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 31 74 d0 10 10 00 |..........1t....| 2023-12-02T03:18:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 fa a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 b5 68 55 15 00 |.hU..| 2023-12-02T03:18:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_8696705f-855b-402d-afd6-1c4d55e06c91_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (92 bytes) : 2023-12-02T03:18:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 1d 13 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fc 9f cc 44 10 00 |.............D..| 2023-12-02T03:18:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 24 00 00 00 06 16 00 00 00 6c 6f 63 |....$........loc| 2023-12-02T03:18:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 61 6c 20 69 6d 61 67 65 20 69 73 20 70 72 69 6d |al image is prim| 2023-12-02T03:18:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 61 72 79 b2 a0 6a 65 7d e9 40 13 00 |ary..je}.@..| 2023-12-02T03:18:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000005c 2023-12-02T03:18:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_8a609a08-21d0-4193-a27d-f581f4516454_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b7 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 f7 70 f7 60 10 00 |...........p.`..| 2023-12-02T03:18:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 e4 a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 f2 1e a5 17 00 |.....| 2023-12-02T03:18:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_8ef4933a-3981-486a-8a90-10311d67bf00_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 c6 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 31 74 d0 10 10 00 |..........1t....| 2023-12-02T03:18:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 fa a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 f2 b6 56 15 00 |..V..| 2023-12-02T03:18:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_92a00385-f1d8-4e86-abb7-0e1810b799c3_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (92 bytes) : 2023-12-02T03:18:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 1d 13 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fc 9f cc 44 10 00 |.............D..| 2023-12-02T03:18:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 24 00 00 00 06 16 00 00 00 6c 6f 63 |....$........loc| 2023-12-02T03:18:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 61 6c 20 69 6d 61 67 65 20 69 73 20 70 72 69 6d |al image is prim| 2023-12-02T03:18:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 61 72 79 b2 a0 6a 65 f7 40 42 13 00 |ary..je.@B..| 2023-12-02T03:18:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000005c 2023-12-02T03:18:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_9333a204-121f-424e-849b-e6aec78f4224_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b7 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 f7 70 f7 60 10 00 |...........p.`..| 2023-12-02T03:18:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 e4 a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 d4 d6 a5 17 00 |.....| 2023-12-02T03:18:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_9802897e-7ccb-4bb2-8ab0-cedc09b0044e_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 e5 13 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 93 cb e2 98 10 00 |................| 2023-12-02T03:18:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 f8 a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 07 2a 9d 12 00 |.*...| 2023-12-02T03:18:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_9b3405db-a7dc-43f6-8462-f7325ff11d39_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 e5 13 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 93 cb e2 98 10 00 |................| 2023-12-02T03:18:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 f8 a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 5c d0 9d 12 00 |\....| 2023-12-02T03:18:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_a30b9861-e056-4151-bcfe-e4a4f2e867dd_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (92 bytes) : 2023-12-02T03:18:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 1d 13 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fc 9f cc 44 10 00 |.............D..| 2023-12-02T03:18:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 24 00 00 00 06 16 00 00 00 6c 6f 63 |....$........loc| 2023-12-02T03:18:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 61 6c 20 69 6d 61 67 65 20 69 73 20 70 72 69 6d |al image is prim| 2023-12-02T03:18:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 61 72 79 b2 a0 6a 65 8d 0f 5e 13 00 |ary..je..^..| 2023-12-02T03:18:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000005c 2023-12-02T03:18:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_a35ab54d-17a7-446d-984d-02da30173d31_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 e5 13 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 93 cb e2 98 10 00 |................| 2023-12-02T03:18:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 f8 a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 a1 6b 9e 12 00 |.k...| 2023-12-02T03:18:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_b629353c-89f1-46c0-b5ff-f652e93d5cb3_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b7 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 f7 70 f7 60 10 00 |...........p.`..| 2023-12-02T03:18:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 e4 a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 aa 91 a6 17 00 |.....| 2023-12-02T03:18:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_b8d1a79d-fa19-4b64-ba6b-5a4658335ba3_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 c6 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 31 74 d0 10 10 00 |..........1t....| 2023-12-02T03:18:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 fa a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 f7 f0 57 15 00 |..W..| 2023-12-02T03:18:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_be8f73e3-45b7-4b7e-8479-7fd35126d17d_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b7 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 f7 70 f7 60 10 00 |...........p.`..| 2023-12-02T03:18:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 e4 a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 11 52 a7 17 00 |.R...| 2023-12-02T03:18:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_c1ab0b04-68aa-4508-ad3f-25906784bf9e_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b7 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 f7 70 f7 60 10 00 |...........p.`..| 2023-12-02T03:18:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 e4 a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 36 09 a8 17 00 |6....| 2023-12-02T03:18:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_c46da9c3-6d9c-4389-a394-a73d56fb3cdf_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b7 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 f7 70 f7 60 10 00 |...........p.`..| 2023-12-02T03:18:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 e4 a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 e7 af a8 17 00 |.....| 2023-12-02T03:18:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_cdf57a21-3ee2-466a-9e94-c4dd4500ddd1_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (92 bytes) : 2023-12-02T03:18:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 1d 13 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fc 9f cc 44 10 00 |.............D..| 2023-12-02T03:18:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 24 00 00 00 06 16 00 00 00 6c 6f 63 |....$........loc| 2023-12-02T03:18:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 61 6c 20 69 6d 61 67 65 20 69 73 20 70 72 69 6d |al image is prim| 2023-12-02T03:18:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 61 72 79 b2 a0 6a 65 af 0f 5f 13 00 |ary..je.._..| 2023-12-02T03:18:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000005c 2023-12-02T03:18:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_d4ceac3d-c782-416e-a7e1-be445a43ae35_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 c6 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 31 74 d0 10 10 00 |..........1t....| 2023-12-02T03:18:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 fa a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 e0 2d 59 15 00 |.-Y..| 2023-12-02T03:18:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_da41f19a-67cd-4518-a939-22e8fe4b6971_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b7 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 f7 70 f7 60 10 00 |...........p.`..| 2023-12-02T03:18:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 e4 a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 18 58 a9 17 00 |.X...| 2023-12-02T03:18:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_ddeae318-0d59-4bf2-b55e-a3f9934ab1b6_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 e5 13 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 93 cb e2 98 10 00 |................| 2023-12-02T03:18:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 f8 a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 ff 03 9f 12 00 |.....| 2023-12-02T03:18:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_e2407e91-d52b-4a70-b3dd-c23362ba73c5_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b7 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 f7 70 f7 60 10 00 |...........p.`..| 2023-12-02T03:18:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 e4 a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 81 10 aa 17 00 |.....| 2023-12-02T03:18:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_e7466612-311c-416f-a7f9-bfa20e74a575_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 e5 13 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 93 cb e2 98 10 00 |................| 2023-12-02T03:18:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 f8 a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 1d 9d 9f 12 00 |.....| 2023-12-02T03:18:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_f12eeb37-6efd-4241-9808-5371a9611fae_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 c6 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 31 74 d0 10 10 00 |..........1t....| 2023-12-02T03:18:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 fa a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 07 5a 5a 15 00 |.ZZ..| 2023-12-02T03:18:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_f1a9b266-fc3d-4246-8a57-fe1695af4b4d_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b7 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 f7 70 f7 60 10 00 |...........p.`..| 2023-12-02T03:18:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 e4 a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 f3 c2 aa 17 00 |.....| 2023-12-02T03:18:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_f71e0710-9762-4b0b-8375-4f15e3c521f1_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 c6 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 31 74 d0 10 10 00 |..........1t....| 2023-12-02T03:18:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 fa a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 91 b7 5b 15 00 |..[..| 2023-12-02T03:18:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_f939d359-e3e3-4881-b4c7-c05064e62969_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 c6 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 31 74 d0 10 10 00 |..........1t....| 2023-12-02T03:18:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 fa a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 e3 03 5d 15 00 |..]..| 2023-12-02T03:18:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_faedaa7c-e337-491d-80ec-3cfa45a9493c_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b7 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 f7 70 f7 60 10 00 |...........p.`..| 2023-12-02T03:18:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 e4 a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 d3 6a ab 17 00 |.j...| 2023-12-02T03:18:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:remote_status_global_ffc4290c-3b4c-45f6-a30f-c93f3cb26275_8bf924be-d648-46b9-b43a-ae071352b043 2023-12-02T03:18:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (85 bytes) : 2023-12-02T03:18:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b7 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 f7 70 f7 60 10 00 |...........p.`..| 2023-12-02T03:18:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-02T03:18:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 e4 a1 6a 65 |rting replay..je| 2023-12-02T03:18:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 2c 09 ac 17 00 |,....| 2023-12-02T03:18:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000055 2023-12-02T03:18:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_03c2b3e5-9969-4a06-b152-32b7163be7b1 2023-12-02T03:18:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.899 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 6d 14 00 00 00 00 00 00 01 |..,....m........| 2023-12-02T03:18:23.899 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 28 77 64 8c 10 00 |..........(wd...| 2023-12-02T03:18:23.899 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.899 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.899 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.899 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.899 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.899 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.899 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.899 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.899 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.899 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.899 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.899 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 33 30 37 2c 22 72 65 6d 6f 74 65 5f 73 6e |86307,"remote_sn| 2023-12-02T03:18:23.899 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 33 30 37 2c 22 72 65 |":1701486307,"re| 2023-12-02T03:18:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d ef a1 6a 65 e0 44 e8 0a 00 |e"}..je.D...| 2023-12-02T03:18:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_09e06fe9-a175-4009-952b-403e63e1ac10 2023-12-02T03:18:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 6d 14 00 00 00 00 00 00 01 |..,....m........| 2023-12-02T03:18:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 28 77 64 8c 10 00 |..........(wd...| 2023-12-02T03:18:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 35 38 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |86581,"remote_sn| 2023-12-02T03:18:23.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 35 38 31 2c 22 72 65 |":1701486581,"re| 2023-12-02T03:18:23.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d ef a1 6a 65 af a9 e9 0a 00 |e"}..je.....| 2023-12-02T03:18:23.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_0a3b7484-3d9a-411e-9ed7-b9ca249bd93a 2023-12-02T03:18:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b8 11 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e8 da f8 be 10 00 |................| 2023-12-02T03:18:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 34 39 30 2c 22 72 65 6d 6f 74 65 5f 73 6e |86490,"remote_sn| 2023-12-02T03:18:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 34 39 30 2c 22 72 65 |":1701486490,"re| 2023-12-02T03:18:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d f9 a1 6a 65 32 73 a4 06 00 |e"}..je2s...| 2023-12-02T03:18:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_0c9ffe84-d24c-4f74-b087-b033f2cc4422 2023-12-02T03:18:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 d0 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 3e 75 19 18 10 00 |..........>u....| 2023-12-02T03:18:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 34 36 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |86461,"remote_sn| 2023-12-02T03:18:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 34 36 31 2c 22 72 65 |":1701486461,"re| 2023-12-02T03:18:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d fd a1 6a 65 4d ac fc 0e 00 |e"}..jeM....| 2023-12-02T03:18:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_14b5094b-8df2-410f-b1dd-ae91594e9848 2023-12-02T03:18:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (300 bytes) : 2023-12-02T03:18:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 d0 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 3e 75 19 18 10 00 |..........>u....| 2023-12-02T03:18:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 31 33 33 37 30 39 38 32 34 2e 30 |hot":133709824.0| 2023-12-02T03:18:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 62 79 74 65 73 22 3a 31 33 33 37 30 39 38 32 34 |bytes":133709824| 2023-12-02T03:18:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 38 36 36 34 34 2c 22 72 65 6d 6f 74 65 5f 73 6e |86644,"remote_sn| 2023-12-02T03:18:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 22 3a 31 37 30 31 34 38 36 36 34 34 2c 22 72 65 |":1701486644,"re| 2023-12-02T03:18:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000120 65 22 7d fd a1 6a 65 64 94 fd 0e 00 |e"}..jed....| 2023-12-02T03:18:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000012c 2023-12-02T03:18:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_19ecaa0e-982b-435c-a3a7-d881b478cee6 2023-12-02T03:18:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 6d 14 00 00 00 00 00 00 01 |..,....m........| 2023-12-02T03:18:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 28 77 64 8c 10 00 |..........(wd...| 2023-12-02T03:18:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 36 33 32 2c 22 72 65 6d 6f 74 65 5f 73 6e |86632,"remote_sn| 2023-12-02T03:18:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 36 33 32 2c 22 72 65 |":1701486632,"re| 2023-12-02T03:18:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d ef a1 6a 65 18 78 ea 0a 00 |e"}..je.x...| 2023-12-02T03:18:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_1d600ef5-05fa-4ec8-bb36-7d891860097c 2023-12-02T03:18:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 6d 14 00 00 00 00 00 00 01 |..,....m........| 2023-12-02T03:18:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 28 77 64 8c 10 00 |..........(wd...| 2023-12-02T03:18:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 36 39 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |86691,"remote_sn| 2023-12-02T03:18:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 36 39 31 2c 22 72 65 |":1701486691,"re| 2023-12-02T03:18:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d ef a1 6a 65 a8 2e eb 0a 00 |e"}..je.....| 2023-12-02T03:18:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_1f4ea1ff-7710-42b2-9726-194a70ed8f1e 2023-12-02T03:18:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b1 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 dc d1 a4 4c 10 00 |.............L..| 2023-12-02T03:18:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 37 31 33 2c 22 72 65 6d 6f 74 65 5f 73 6e |86713,"remote_sn| 2023-12-02T03:18:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 37 31 33 2c 22 72 65 |":1701486713,"re| 2023-12-02T03:18:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d e5 a1 6a 65 2e a1 61 0f 00 |e"}..je..a..| 2023-12-02T03:18:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_258c1b03-48a5-42f2-8745-ac0283c8bc87 2023-12-02T03:18:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (300 bytes) : 2023-12-02T03:18:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b8 11 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e8 da f8 be 10 00 |................| 2023-12-02T03:18:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 31 33 34 30 35 39 30 30 38 2e 30 |hot":134059008.0| 2023-12-02T03:18:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 62 79 74 65 73 22 3a 31 33 34 30 35 39 30 30 38 |bytes":134059008| 2023-12-02T03:18:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 38 36 36 37 30 2c 22 72 65 6d 6f 74 65 5f 73 6e |86670,"remote_sn| 2023-12-02T03:18:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 22 3a 31 37 30 31 34 38 36 36 37 30 2c 22 72 65 |":1701486670,"re| 2023-12-02T03:18:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000120 65 22 7d f9 a1 6a 65 a5 5b a5 06 00 |e"}..je.[...| 2023-12-02T03:18:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000012c 2023-12-02T03:18:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_26af4e3d-22a4-46d1-af96-3f3836056461 2023-12-02T03:18:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (300 bytes) : 2023-12-02T03:18:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 d0 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 3e 75 19 18 10 00 |..........>u....| 2023-12-02T03:18:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 31 33 34 30 33 32 33 38 34 2e 30 |hot":134032384.0| 2023-12-02T03:18:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 62 79 74 65 73 22 3a 31 33 34 30 33 32 33 38 34 |bytes":134032384| 2023-12-02T03:18:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 38 36 37 38 39 2c 22 72 65 6d 6f 74 65 5f 73 6e |86789,"remote_sn| 2023-12-02T03:18:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 22 3a 31 37 30 31 34 38 36 37 38 39 2c 22 72 65 |":1701486789,"re| 2023-12-02T03:18:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000120 65 22 7d fd a1 6a 65 07 33 fe 0e 00 |e"}..je.3...| 2023-12-02T03:18:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000012c 2023-12-02T03:18:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_2e4177a2-c078-4cd9-b5d2-ee2477a02290 2023-12-02T03:18:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 d0 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 3e 75 19 18 10 00 |..........>u....| 2023-12-02T03:18:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 34 37 32 2c 22 72 65 6d 6f 74 65 5f 73 6e |86472,"remote_sn| 2023-12-02T03:18:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 34 37 32 2c 22 72 65 |":1701486472,"re| 2023-12-02T03:18:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d fd a1 6a 65 85 da fe 0e 00 |e"}..je.....| 2023-12-02T03:18:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_3d00c370-35ca-4837-a707-0aa3e5b4c7f4 2023-12-02T03:18:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 6d 14 00 00 00 00 00 00 01 |..,....m........| 2023-12-02T03:18:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 28 77 64 8c 10 00 |..........(wd...| 2023-12-02T03:18:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 32 39 34 2c 22 72 65 6d 6f 74 65 5f 73 6e |86294,"remote_sn| 2023-12-02T03:18:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 32 39 34 2c 22 72 65 |":1701486294,"re| 2023-12-02T03:18:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d ef a1 6a 65 a6 e5 eb 0a 00 |e"}..je.....| 2023-12-02T03:18:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_453a5e01-39a4-415e-bf21-d59460038eff 2023-12-02T03:18:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 6d 14 00 00 00 00 00 00 01 |..,....m........| 2023-12-02T03:18:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 28 77 64 8c 10 00 |..........(wd...| 2023-12-02T03:18:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 32 37 30 2c 22 72 65 6d 6f 74 65 5f 73 6e |86270,"remote_sn| 2023-12-02T03:18:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 32 37 30 2c 22 72 65 |":1701486270,"re| 2023-12-02T03:18:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d ef a1 6a 65 7c da ec 0a 00 |e"}..je|....| 2023-12-02T03:18:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_49c16484-3472-4671-8be8-03c1846564a3 2023-12-02T03:18:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 6d 14 00 00 00 00 00 00 01 |..,....m........| 2023-12-02T03:18:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 28 77 64 8c 10 00 |..........(wd...| 2023-12-02T03:18:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 35 37 33 2c 22 72 65 6d 6f 74 65 5f 73 6e |86573,"remote_sn| 2023-12-02T03:18:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 35 37 33 2c 22 72 65 |":1701486573,"re| 2023-12-02T03:18:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d ef a1 6a 65 22 9b ed 0a 00 |e"}..je"....| 2023-12-02T03:18:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_4f77376e-57d2-4b09-b376-6129ff7390b9 2023-12-02T03:18:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 d0 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 3e 75 19 18 10 00 |..........>u....| 2023-12-02T03:18:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 35 31 33 2c 22 72 65 6d 6f 74 65 5f 73 6e |86513,"remote_sn| 2023-12-02T03:18:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 35 31 33 2c 22 72 65 |":1701486513,"re| 2023-12-02T03:18:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d fd a1 6a 65 69 6c ff 0e 00 |e"}..jeil...| 2023-12-02T03:18:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_55cf3916-f72c-47f2-b9d9-b3f8b8496445 2023-12-02T03:18:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 d0 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 3e 75 19 18 10 00 |..........>u....| 2023-12-02T03:18:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 34 34 38 2c 22 72 65 6d 6f 74 65 5f 73 6e |86448,"remote_sn| 2023-12-02T03:18:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 34 34 38 2c 22 72 65 |":1701486448,"re| 2023-12-02T03:18:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d fd a1 6a 65 07 09 00 0f 00 |e"}..je.....| 2023-12-02T03:18:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_56d75e57-f434-4172-81f2-851edf5c5eb5 2023-12-02T03:18:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 6d 14 00 00 00 00 00 00 01 |..,....m........| 2023-12-02T03:18:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 28 77 64 8c 10 00 |..........(wd...| 2023-12-02T03:18:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 33 33 39 2c 22 72 65 6d 6f 74 65 5f 73 6e |86339,"remote_sn| 2023-12-02T03:18:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 33 33 39 2c 22 72 65 |":1701486339,"re| 2023-12-02T03:18:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d ef a1 6a 65 e9 4d ee 0a 00 |e"}..je.M...| 2023-12-02T03:18:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_5a34854e-65bf-43d7-9d2c-e0481cc5a159 2023-12-02T03:18:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b8 11 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e8 da f8 be 10 00 |................| 2023-12-02T03:18:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 36 39 39 2c 22 72 65 6d 6f 74 65 5f 73 6e |86699,"remote_sn| 2023-12-02T03:18:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 36 39 39 2c 22 72 65 |":1701486699,"re| 2023-12-02T03:18:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d f9 a1 6a 65 36 04 a6 06 00 |e"}..je6....| 2023-12-02T03:18:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_5c374561-6563-424e-bb48-6c5014bde1fc 2023-12-02T03:18:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 d0 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 3e 75 19 18 10 00 |..........>u....| 2023-12-02T03:18:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 33 37 37 2c 22 72 65 6d 6f 74 65 5f 73 6e |86377,"remote_sn| 2023-12-02T03:18:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 33 37 37 2c 22 72 65 |":1701486377,"re| 2023-12-02T03:18:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d fd a1 6a 65 44 9b 00 0f 00 |e"}..jeD....| 2023-12-02T03:18:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_5db3178e-6670-496c-bd76-373d30e83937 2023-12-02T03:18:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 6d 14 00 00 00 00 00 00 01 |..,....m........| 2023-12-02T03:18:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 28 77 64 8c 10 00 |..........(wd...| 2023-12-02T03:18:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.930 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 34 34 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |86441,"remote_sn| 2023-12-02T03:18:23.930 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.930 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 34 34 31 2c 22 72 65 |":1701486441,"re| 2023-12-02T03:18:23.930 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.930 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d ef a1 6a 65 6e fd ee 0a 00 |e"}..jen....| 2023-12-02T03:18:23.930 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.930 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.930 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_730af9f3-f6b6-4cf2-9474-5684c31f584d 2023-12-02T03:18:23.930 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.930 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b1 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.930 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 dc d1 a4 4c 10 00 |.............L..| 2023-12-02T03:18:23.930 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.930 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.930 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 36 32 36 2c 22 72 65 6d 6f 74 65 5f 73 6e |86626,"remote_sn| 2023-12-02T03:18:23.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 36 32 36 2c 22 72 65 |":1701486626,"re| 2023-12-02T03:18:23.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d e5 a1 6a 65 f8 d3 62 0f 00 |e"}..je..b..| 2023-12-02T03:18:23.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.931 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_7dc110f7-a667-4d4c-9e52-4c6fe497aadb 2023-12-02T03:18:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b8 11 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e8 da f8 be 10 00 |................| 2023-12-02T03:18:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 32 36 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |86261,"remote_sn| 2023-12-02T03:18:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 32 36 31 2c 22 72 65 |":1701486261,"re| 2023-12-02T03:18:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d f9 a1 6a 65 d8 a7 a6 06 00 |e"}..je.....| 2023-12-02T03:18:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_807f0ce9-67ef-4e3a-987e-bfe34f97a832 2023-12-02T03:18:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 6d 14 00 00 00 00 00 00 01 |..,....m........| 2023-12-02T03:18:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 28 77 64 8c 10 00 |..........(wd...| 2023-12-02T03:18:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 37 30 35 2c 22 72 65 6d 6f 74 65 5f 73 6e |86705,"remote_sn| 2023-12-02T03:18:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 37 30 35 2c 22 72 65 |":1701486705,"re| 2023-12-02T03:18:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d ef a1 6a 65 38 ae ef 0a 00 |e"}..je8....| 2023-12-02T03:18:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_8117a00a-84c1-4b06-96bd-60840c20cce4 2023-12-02T03:18:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 6d 14 00 00 00 00 00 00 01 |..,....m........| 2023-12-02T03:18:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 28 77 64 8c 10 00 |..........(wd...| 2023-12-02T03:18:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 37 33 32 2c 22 72 65 6d 6f 74 65 5f 73 6e |86732,"remote_sn| 2023-12-02T03:18:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 37 33 32 2c 22 72 65 |":1701486732,"re| 2023-12-02T03:18:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d ef a1 6a 65 ac 58 f0 0a 00 |e"}..je.X...| 2023-12-02T03:18:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_8696705f-855b-402d-afd6-1c4d55e06c91 2023-12-02T03:18:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (300 bytes) : 2023-12-02T03:18:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b8 11 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e8 da f8 be 10 00 |................| 2023-12-02T03:18:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 31 33 33 32 34 38 30 30 30 2e 30 |hot":133248000.0| 2023-12-02T03:18:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 62 79 74 65 73 22 3a 31 33 33 32 34 38 30 30 30 |bytes":133248000| 2023-12-02T03:18:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 38 36 37 35 32 2c 22 72 65 6d 6f 74 65 5f 73 6e |86752,"remote_sn| 2023-12-02T03:18:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 22 3a 31 37 30 31 34 38 36 37 35 32 2c 22 72 65 |":1701486752,"re| 2023-12-02T03:18:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000120 65 22 7d f9 a1 6a 65 f1 3e a7 06 00 |e"}..je.>...| 2023-12-02T03:18:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000012c 2023-12-02T03:18:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_8a609a08-21d0-4193-a27d-f581f4516454 2023-12-02T03:18:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b1 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 dc d1 a4 4c 10 00 |.............L..| 2023-12-02T03:18:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 34 30 35 2c 22 72 65 6d 6f 74 65 5f 73 6e |86405,"remote_sn| 2023-12-02T03:18:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 34 30 35 2c 22 72 65 |":1701486405,"re| 2023-12-02T03:18:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d e5 a1 6a 65 a3 9b 63 0f 00 |e"}..je..c..| 2023-12-02T03:18:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_8ef4933a-3981-486a-8a90-10311d67bf00 2023-12-02T03:18:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (300 bytes) : 2023-12-02T03:18:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b1 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 dc d1 a4 4c 10 00 |.............L..| 2023-12-02T03:18:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 31 33 34 30 39 32 38 30 30 2e 30 |hot":134092800.0| 2023-12-02T03:18:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 62 79 74 65 73 22 3a 31 33 34 30 39 32 38 30 30 |bytes":134092800| 2023-12-02T03:18:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 38 36 37 34 36 2c 22 72 65 6d 6f 74 65 5f 73 6e |86746,"remote_sn| 2023-12-02T03:18:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 22 3a 31 37 30 31 34 38 36 37 34 36 2c 22 72 65 |":1701486746,"re| 2023-12-02T03:18:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000120 65 22 7d e5 a1 6a 65 65 68 64 0f 00 |e"}..jeehd..| 2023-12-02T03:18:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000012c 2023-12-02T03:18:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_92a00385-f1d8-4e86-abb7-0e1810b799c3 2023-12-02T03:18:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b8 11 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e8 da f8 be 10 00 |................| 2023-12-02T03:18:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 35 31 39 2c 22 72 65 6d 6f 74 65 5f 73 6e |86519,"remote_sn| 2023-12-02T03:18:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 35 31 39 2c 22 72 65 |":1701486519,"re| 2023-12-02T03:18:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d f9 a1 6a 65 5c dd a7 06 00 |e"}..je\....| 2023-12-02T03:18:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_9333a204-121f-424e-849b-e6aec78f4224 2023-12-02T03:18:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b8 11 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e8 da f8 be 10 00 |................| 2023-12-02T03:18:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 33 39 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |86391,"remote_sn| 2023-12-02T03:18:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 33 39 31 2c 22 72 65 |":1701486391,"re| 2023-12-02T03:18:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d f9 a1 6a 65 28 75 a8 06 00 |e"}..je(u...| 2023-12-02T03:18:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_9802897e-7ccb-4bb2-8ab0-cedc09b0044e 2023-12-02T03:18:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b1 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 dc d1 a4 4c 10 00 |.............L..| 2023-12-02T03:18:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 33 36 36 2c 22 72 65 6d 6f 74 65 5f 73 6e |86366,"remote_sn| 2023-12-02T03:18:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 33 36 36 2c 22 72 65 |":1701486366,"re| 2023-12-02T03:18:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d e5 a1 6a 65 a4 65 65 0f 00 |e"}..je.ee..| 2023-12-02T03:18:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_9b3405db-a7dc-43f6-8462-f7325ff11d39 2023-12-02T03:18:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (300 bytes) : 2023-12-02T03:18:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 d0 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 3e 75 19 18 10 00 |..........>u....| 2023-12-02T03:18:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 31 33 33 38 33 30 36 35 36 2e 30 |hot":133830656.0| 2023-12-02T03:18:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 62 79 74 65 73 22 3a 31 33 33 38 33 30 36 35 36 |bytes":133830656| 2023-12-02T03:18:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 38 36 37 38 33 2c 22 72 65 6d 6f 74 65 5f 73 6e |86783,"remote_sn| 2023-12-02T03:18:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 22 3a 31 37 30 31 34 38 36 37 38 33 2c 22 72 65 |":1701486783,"re| 2023-12-02T03:18:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000120 65 22 7d fd a1 6a 65 53 38 01 0f 00 |e"}..jeS8...| 2023-12-02T03:18:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000012c 2023-12-02T03:18:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_a30b9861-e056-4151-bcfe-e4a4f2e867dd 2023-12-02T03:18:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (300 bytes) : 2023-12-02T03:18:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 d0 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 3e 75 19 18 10 00 |..........>u....| 2023-12-02T03:18:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 31 33 34 31 38 39 30 35 36 2e 30 |hot":134189056.0| 2023-12-02T03:18:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 62 79 74 65 73 22 3a 31 33 34 31 38 39 30 35 36 |bytes":134189056| 2023-12-02T03:18:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 38 36 36 37 38 2c 22 72 65 6d 6f 74 65 5f 73 6e |86678,"remote_sn| 2023-12-02T03:18:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 22 3a 31 37 30 31 34 38 36 36 37 38 2c 22 72 65 |":1701486678,"re| 2023-12-02T03:18:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000120 65 22 7d fd a1 6a 65 6b d1 01 0f 00 |e"}..jek....| 2023-12-02T03:18:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000012c 2023-12-02T03:18:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_a35ab54d-17a7-446d-984d-02da30173d31 2023-12-02T03:18:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (300 bytes) : 2023-12-02T03:18:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 d0 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 3e 75 19 18 10 00 |..........>u....| 2023-12-02T03:18:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 31 33 33 37 33 39 35 32 30 2e 30 |hot":133739520.0| 2023-12-02T03:18:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 62 79 74 65 73 22 3a 31 33 33 37 33 39 35 32 30 |bytes":133739520| 2023-12-02T03:18:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 38 36 35 38 37 2c 22 72 65 6d 6f 74 65 5f 73 6e |86587,"remote_sn| 2023-12-02T03:18:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 22 3a 31 37 30 31 34 38 36 35 38 37 2c 22 72 65 |":1701486587,"re| 2023-12-02T03:18:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000120 65 22 7d fd a1 6a 65 c1 61 02 0f 00 |e"}..je.a...| 2023-12-02T03:18:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000012c 2023-12-02T03:18:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_b629353c-89f1-46c0-b5ff-f652e93d5cb3 2023-12-02T03:18:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 6d 14 00 00 00 00 00 00 01 |..,....m........| 2023-12-02T03:18:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 28 77 64 8c 10 00 |..........(wd...| 2023-12-02T03:18:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 37 34 30 2c 22 72 65 6d 6f 74 65 5f 73 6e |86740,"remote_sn| 2023-12-02T03:18:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 37 34 30 2c 22 72 65 |":1701486740,"re| 2023-12-02T03:18:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d ef a1 6a 65 e2 01 f1 0a 00 |e"}..je.....| 2023-12-02T03:18:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_b8d1a79d-fa19-4b64-ba6b-5a4658335ba3 2023-12-02T03:18:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (300 bytes) : 2023-12-02T03:18:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 d0 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 3e 75 19 18 10 00 |..........>u....| 2023-12-02T03:18:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 31 33 34 31 33 32 37 33 36 2e 30 |hot":134132736.0| 2023-12-02T03:18:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 62 79 74 65 73 22 3a 31 33 34 31 33 32 37 33 36 |bytes":134132736| 2023-12-02T03:18:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 38 36 37 31 39 2c 22 72 65 6d 6f 74 65 5f 73 6e |86719,"remote_sn| 2023-12-02T03:18:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 22 3a 31 37 30 31 34 38 36 37 31 39 2c 22 72 65 |":1701486719,"re| 2023-12-02T03:18:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000120 65 22 7d fd a1 6a 65 9a f0 02 0f 00 |e"}..je.....| 2023-12-02T03:18:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000012c 2023-12-02T03:18:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_be8f73e3-45b7-4b7e-8479-7fd35126d17d 2023-12-02T03:18:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b1 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 dc d1 a4 4c 10 00 |.............L..| 2023-12-02T03:18:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 35 30 34 2c 22 72 65 6d 6f 74 65 5f 73 6e |86504,"remote_sn| 2023-12-02T03:18:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 35 30 34 2c 22 72 65 |":1701486504,"re| 2023-12-02T03:18:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d e5 a1 6a 65 f3 18 66 0f 00 |e"}..je..f..| 2023-12-02T03:18:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_c1ab0b04-68aa-4508-ad3f-25906784bf9e 2023-12-02T03:18:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b1 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 dc d1 a4 4c 10 00 |.............L..| 2023-12-02T03:18:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 34 31 36 2c 22 72 65 6d 6f 74 65 5f 73 6e |86416,"remote_sn| 2023-12-02T03:18:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 34 31 36 2c 22 72 65 |":1701486416,"re| 2023-12-02T03:18:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d e5 a1 6a 65 fb ce 66 0f 00 |e"}..je..f..| 2023-12-02T03:18:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_c46da9c3-6d9c-4389-a394-a73d56fb3cdf 2023-12-02T03:18:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b8 11 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e8 da f8 be 10 00 |................| 2023-12-02T03:18:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 34 37 39 2c 22 72 65 6d 6f 74 65 5f 73 6e |86479,"remote_sn| 2023-12-02T03:18:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 34 37 39 2c 22 72 65 |":1701486479,"re| 2023-12-02T03:18:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d f9 a1 6a 65 41 08 a9 06 00 |e"}..jeA....| 2023-12-02T03:18:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_cdf57a21-3ee2-466a-9e94-c4dd4500ddd1 2023-12-02T03:18:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (300 bytes) : 2023-12-02T03:18:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b1 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 dc d1 a4 4c 10 00 |.............L..| 2023-12-02T03:18:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 31 33 34 31 37 39 38 34 30 2e 30 |hot":134179840.0| 2023-12-02T03:18:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 62 79 74 65 73 22 3a 31 33 34 31 37 39 38 34 30 |bytes":134179840| 2023-12-02T03:18:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 38 36 36 38 33 2c 22 72 65 6d 6f 74 65 5f 73 6e |86683,"remote_sn| 2023-12-02T03:18:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 22 3a 31 37 30 31 34 38 36 36 38 33 2c 22 72 65 |":1701486683,"re| 2023-12-02T03:18:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000120 65 22 7d e5 a1 6a 65 de 83 67 0f 00 |e"}..je..g..| 2023-12-02T03:18:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000012c 2023-12-02T03:18:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_d4ceac3d-c782-416e-a7e1-be445a43ae35 2023-12-02T03:18:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b8 11 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e8 da f8 be 10 00 |................| 2023-12-02T03:18:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 33 32 38 2c 22 72 65 6d 6f 74 65 5f 73 6e |86328,"remote_sn| 2023-12-02T03:18:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 33 32 38 2c 22 72 65 |":1701486328,"re| 2023-12-02T03:18:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d f9 a1 6a 65 ac 9f a9 06 00 |e"}..je.....| 2023-12-02T03:18:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_da41f19a-67cd-4518-a939-22e8fe4b6971 2023-12-02T03:18:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b8 11 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e8 da f8 be 10 00 |................| 2023-12-02T03:18:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 36 35 36 2c 22 72 65 6d 6f 74 65 5f 73 6e |86656,"remote_sn| 2023-12-02T03:18:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 36 35 36 2c 22 72 65 |":1701486656,"re| 2023-12-02T03:18:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d f9 a1 6a 65 cd 2f aa 06 00 |e"}..je./...| 2023-12-02T03:18:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_ddeae318-0d59-4bf2-b55e-a3f9934ab1b6 2023-12-02T03:18:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (300 bytes) : 2023-12-02T03:18:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 d0 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 3e 75 19 18 10 00 |..........>u....| 2023-12-02T03:18:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 31 33 34 30 31 37 30 32 34 2e 30 |hot":134017024.0| 2023-12-02T03:18:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 62 79 74 65 73 22 3a 31 33 34 30 31 37 30 32 34 |bytes":134017024| 2023-12-02T03:18:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 38 36 36 36 32 2c 22 72 65 6d 6f 74 65 5f 73 6e |86662,"remote_sn| 2023-12-02T03:18:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 22 3a 31 37 30 31 34 38 36 36 36 32 2c 22 72 65 |":1701486662,"re| 2023-12-02T03:18:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000120 65 22 7d fd a1 6a 65 f8 82 03 0f 00 |e"}..je.....| 2023-12-02T03:18:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000012c 2023-12-02T03:18:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_e2407e91-d52b-4a70-b3dd-c23362ba73c5 2023-12-02T03:18:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b8 11 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e8 da f8 be 10 00 |................| 2023-12-02T03:18:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 33 35 33 2c 22 72 65 6d 6f 74 65 5f 73 6e |86353,"remote_sn| 2023-12-02T03:18:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 33 35 33 2c 22 72 65 |":1701486353,"re| 2023-12-02T03:18:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d f9 a1 6a 65 9f cd aa 06 00 |e"}..je.....| 2023-12-02T03:18:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_e7466612-311c-416f-a7f9-bfa20e74a575 2023-12-02T03:18:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b1 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 dc d1 a4 4c 10 00 |.............L..| 2023-12-02T03:18:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 32 38 33 2c 22 72 65 6d 6f 74 65 5f 73 6e |86283,"remote_sn| 2023-12-02T03:18:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 32 38 33 2c 22 72 65 |":1701486283,"re| 2023-12-02T03:18:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d e5 a1 6a 65 24 3f 68 0f 00 |e"}..je$?h..| 2023-12-02T03:18:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_f12eeb37-6efd-4241-9808-5371a9611fae 2023-12-02T03:18:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b1 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 dc d1 a4 4c 10 00 |.............L..| 2023-12-02T03:18:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 34 32 39 2c 22 72 65 6d 6f 74 65 5f 73 6e |86429,"remote_sn| 2023-12-02T03:18:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 34 32 39 2c 22 72 65 |":1701486429,"re| 2023-12-02T03:18:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d e5 a1 6a 65 8f f9 68 0f 00 |e"}..je..h..| 2023-12-02T03:18:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_f1a9b266-fc3d-4246-8a57-fe1695af4b4d 2023-12-02T03:18:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (300 bytes) : 2023-12-02T03:18:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b8 11 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e8 da f8 be 10 00 |................| 2023-12-02T03:18:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 31 33 34 32 30 35 34 34 30 2e 30 |hot":134205440.0| 2023-12-02T03:18:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 62 79 74 65 73 22 3a 31 33 34 32 30 35 34 34 30 |bytes":134205440| 2023-12-02T03:18:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 38 36 36 33 38 2c 22 72 65 6d 6f 74 65 5f 73 6e |86638,"remote_sn| 2023-12-02T03:18:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 22 3a 31 37 30 31 34 38 36 36 33 38 2c 22 72 65 |":1701486638,"re| 2023-12-02T03:18:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000120 65 22 7d f9 a1 6a 65 26 6b ab 06 00 |e"}..je&k...| 2023-12-02T03:18:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000012c 2023-12-02T03:18:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_f71e0710-9762-4b0b-8375-4f15e3c521f1 2023-12-02T03:18:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b1 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 dc d1 a4 4c 10 00 |.............L..| 2023-12-02T03:18:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 36 35 30 2c 22 72 65 6d 6f 74 65 5f 73 6e |86650,"remote_sn| 2023-12-02T03:18:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 36 35 30 2c 22 72 65 |":1701486650,"re| 2023-12-02T03:18:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d e5 a1 6a 65 04 9d 69 0f 00 |e"}..je..i..| 2023-12-02T03:18:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_f939d359-e3e3-4881-b4c7-c05064e62969 2023-12-02T03:18:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b8 11 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e8 da f8 be 10 00 |................| 2023-12-02T03:18:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 35 33 34 2c 22 72 65 6d 6f 74 65 5f 73 6e |86534,"remote_sn| 2023-12-02T03:18:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 35 33 34 2c 22 72 65 |":1701486534,"re| 2023-12-02T03:18:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d f9 a1 6a 65 d0 01 ac 06 00 |e"}..je.....| 2023-12-02T03:18:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_faedaa7c-e337-491d-80ec-3cfa45a9493c 2023-12-02T03:18:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (300 bytes) : 2023-12-02T03:18:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b1 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 dc d1 a4 4c 10 00 |.............L..| 2023-12-02T03:18:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 31 33 33 36 35 36 35 37 36 2e 30 |hot":133656576.0| 2023-12-02T03:18:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 62 79 74 65 73 22 3a 31 33 33 36 35 36 35 37 36 |bytes":133656576| 2023-12-02T03:18:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 38 36 37 32 35 2c 22 72 65 6d 6f 74 65 5f 73 6e |86725,"remote_sn| 2023-12-02T03:18:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 22 3a 31 37 30 31 34 38 36 37 32 35 2c 22 72 65 |":1701486725,"re| 2023-12-02T03:18:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000120 65 22 7d e5 a1 6a 65 6e 44 6a 0f 00 |e"}..jenDj..| 2023-12-02T03:18:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000012c 2023-12-02T03:18:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:status_global_ffc4290c-3b4c-45f6-a30f-c93f3cb26275 2023-12-02T03:18:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (284 bytes) : 2023-12-02T03:18:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 2c 00 00 00 08 b1 12 00 00 00 00 00 00 01 |..,.............| 2023-12-02T03:18:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 dc d1 a4 4c 10 00 |.............L..| 2023-12-02T03:18:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 02 00 00 00 ac 15 0f 39 00 00 00 00 00 00 |.........9......| 2023-12-02T03:18:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-02T03:18:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-02T03:18:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-02T03:18:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-02T03:18:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-02T03:18:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-02T03:18:23.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-02T03:18:23.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-02T03:18:23.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-02T03:18:23.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 34 |timestamp":17014| 2023-12-02T03:18:23.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000d0 38 36 33 31 33 2c 22 72 65 6d 6f 74 65 5f 73 6e |86313,"remote_sn| 2023-12-02T03:18:23.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-02T03:18:23.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:000000f0 22 3a 31 37 30 31 34 38 36 33 31 33 2c 22 72 65 |":1701486313,"re| 2023-12-02T03:18:23.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-02T03:18:23.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000110 65 22 7d e5 a1 6a 65 f5 fa 6a 0f 00 |e"}..je..j..| 2023-12-02T03:18:23.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:0000011c 2023-12-02T03:18:23.977 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster1 mirror rbd_mirror_leader omap vals' 2023-12-02T03:18:23.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster1 mirror rbd_mirror_leader omap vals 2023-12-02T03:18:23.978 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rados --cluster cluster1 -p mirror --namespace '' listomapvals rbd_mirror_leader 2023-12-02T03:18:23.986 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_03c2b3e5-9969-4a06-b152-32b7163be7b1 2023-12-02T03:18:23.986 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 32 32 39 b8 a0 |..........5229..| 2023-12-02T03:18:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 d0 21 59 0b 0a 00 00 00 01 01 04 00 00 00 |je.!Y...........| 2023-12-02T03:18:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_09e06fe9-a175-4009-952b-403e63e1ac10 2023-12-02T03:18:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 32 32 39 b8 a0 |..........5229..| 2023-12-02T03:18:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 63 4f 59 0b 0a 00 00 00 01 01 04 00 00 00 |jecOY...........| 2023-12-02T03:18:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_0a3b7484-3d9a-411e-9ed7-b9ca249bd93a 2023-12-02T03:18:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 35 33 36 b8 a0 |..........4536..| 2023-12-02T03:18:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 4f 6d 59 0b 0a 00 00 00 01 01 04 00 00 00 |jeOmY...........| 2023-12-02T03:18:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_0c9ffe84-d24c-4f74-b087-b033f2cc4422 2023-12-02T03:18:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 31 36 7e 9f |..........4816~.| 2023-12-02T03:18:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 aa 87 96 32 0a 00 00 00 01 01 04 00 00 00 |je...2..........| 2023-12-02T03:18:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_14b5094b-8df2-410f-b1dd-ae91594e9848 2023-12-02T03:18:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 31 36 80 9f |..........4816..| 2023-12-02T03:18:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 46 a0 c8 32 0a 00 00 00 01 01 04 00 00 00 |jeF..2..........| 2023-12-02T03:18:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_19ecaa0e-982b-435c-a3a7-d881b478cee6 2023-12-02T03:18:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 32 32 39 a9 a0 |..........5229..| 2023-12-02T03:18:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 af df 98 0c 0a 00 00 00 01 01 04 00 00 00 |je..............| 2023-12-02T03:18:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_1d600ef5-05fa-4ec8-bb36-7d891860097c 2023-12-02T03:18:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 32 32 39 a9 a0 |..........5229..| 2023-12-02T03:18:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 a7 0a 99 0c 0a 00 00 00 01 01 04 00 00 00 |je..............| 2023-12-02T03:18:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_1f4ea1ff-7710-42b2-9726-194a70ed8f1e 2023-12-02T03:18:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 38 35 b8 a0 |..........4785..| 2023-12-02T03:18:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 fa 87 59 0b 0a 00 00 00 01 01 04 00 00 00 |je..Y...........| 2023-12-02T03:18:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_258c1b03-48a5-42f2-8745-ac0283c8bc87 2023-12-02T03:18:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 35 33 36 80 9f |..........4536..| 2023-12-02T03:18:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 fe bf c8 32 0a 00 00 00 01 01 04 00 00 00 |je...2..........| 2023-12-02T03:18:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_26af4e3d-22a4-46d1-af96-3f3836056461 2023-12-02T03:18:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 31 36 b8 a0 |..........4816..| 2023-12-02T03:18:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 90 b1 59 0b 0a 00 00 00 01 01 04 00 00 00 |je..Y...........| 2023-12-02T03:18:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_2e4177a2-c078-4cd9-b5d2-ee2477a02290 2023-12-02T03:18:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 31 36 89 9f |..........4816..| 2023-12-02T03:18:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 e7 42 30 04 0a 00 00 00 01 01 04 00 00 00 |je.B0...........| 2023-12-02T03:18:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_3d00c370-35ca-4837-a707-0aa3e5b4c7f4 2023-12-02T03:18:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 32 32 39 a9 a0 |..........5229..| 2023-12-02T03:18:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 7b 2c 99 0c 0a 00 00 00 01 01 04 00 00 00 |je{,............| 2023-12-02T03:18:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_453a5e01-39a4-415e-bf21-d59460038eff 2023-12-02T03:18:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 32 32 39 a9 a0 |..........5229..| 2023-12-02T03:18:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 83 46 99 0c 0a 00 00 00 01 01 04 00 00 00 |je.F............| 2023-12-02T03:18:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_49c16484-3472-4671-8be8-03c1846564a3 2023-12-02T03:18:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 32 32 39 b8 a0 |..........5229..| 2023-12-02T03:18:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 81 27 5a 0b 0a 00 00 00 01 01 04 00 00 00 |je.'Z...........| 2023-12-02T03:18:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_4f77376e-57d2-4b09-b376-6129ff7390b9 2023-12-02T03:18:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 31 36 b2 9f |..........4816..| 2023-12-02T03:18:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 9f 4e d2 0c 0a 00 00 00 01 01 04 00 00 00 |je.N............| 2023-12-02T03:18:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_55cf3916-f72c-47f2-b9d9-b3f8b8496445 2023-12-02T03:18:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 31 36 71 9f |..........4816q.| 2023-12-02T03:18:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 c4 89 c4 24 0a 00 00 00 01 01 04 00 00 00 |je...$..........| 2023-12-02T03:18:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_56d75e57-f434-4172-81f2-851edf5c5eb5 2023-12-02T03:18:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 32 32 39 a9 a0 |..........5229..| 2023-12-02T03:18:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 c2 64 99 0c 0a 00 00 00 01 01 04 00 00 00 |je.d............| 2023-12-02T03:18:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_5a34854e-65bf-43d7-9d2c-e0481cc5a159 2023-12-02T03:18:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 35 33 36 b8 a0 |..........4536..| 2023-12-02T03:18:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 3e 58 5a 0b 0a 00 00 00 01 01 04 00 00 00 |je>XZ...........| 2023-12-02T03:18:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_5c374561-6563-424e-bb48-6c5014bde1fc 2023-12-02T03:18:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 31 36 80 9f |..........4816..| 2023-12-02T03:18:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 b5 ec c8 32 0a 00 00 00 01 01 04 00 00 00 |je...2..........| 2023-12-02T03:18:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_5db3178e-6670-496c-bd76-373d30e83937 2023-12-02T03:18:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 32 32 39 a9 a0 |..........5229..| 2023-12-02T03:18:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 a8 7f 99 0c 0a 00 00 00 01 01 04 00 00 00 |je..............| 2023-12-02T03:18:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_730af9f3-f6b6-4cf2-9474-5684c31f584d 2023-12-02T03:18:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 38 35 b8 a0 |..........4785..| 2023-12-02T03:18:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 75 7e 5a 0b 0a 00 00 00 01 01 04 00 00 00 |jeu~Z...........| 2023-12-02T03:18:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_7dc110f7-a667-4d4c-9e52-4c6fe497aadb 2023-12-02T03:18:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 35 33 36 18 a0 |..........4536..| 2023-12-02T03:18:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 d0 68 78 08 0a 00 00 00 01 01 04 00 00 00 |je.hx...........| 2023-12-02T03:18:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_807f0ce9-67ef-4e3a-987e-bfe34f97a832 2023-12-02T03:18:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 32 32 39 a9 a0 |..........5229..| 2023-12-02T03:18:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 3a 97 99 0c 0a 00 00 00 01 01 04 00 00 00 |je:.............| 2023-12-02T03:18:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_8117a00a-84c1-4b06-96bd-60840c20cce4 2023-12-02T03:18:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 32 32 39 a9 a0 |..........5229..| 2023-12-02T03:18:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 81 ae 99 0c 0a 00 00 00 01 01 04 00 00 00 |je..............| 2023-12-02T03:18:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_8696705f-855b-402d-afd6-1c4d55e06c91 2023-12-02T03:18:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 35 33 36 80 9f |..........4536..| 2023-12-02T03:18:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 92 01 c9 32 0a 00 00 00 01 01 04 00 00 00 |je...2..........| 2023-12-02T03:18:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_8a609a08-21d0-4193-a27d-f581f4516454 2023-12-02T03:18:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 38 35 18 a0 |..........4785..| 2023-12-02T03:18:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 89 93 78 08 0a 00 00 00 01 01 04 00 00 00 |je..x...........| 2023-12-02T03:18:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_8ef4933a-3981-486a-8a90-10311d67bf00 2023-12-02T03:18:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 38 35 a5 9e |..........4785..| 2023-12-02T03:18:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 bd f4 50 0a 0a 00 00 00 01 01 04 00 00 00 |je..P...........| 2023-12-02T03:18:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_92a00385-f1d8-4e86-abb7-0e1810b799c3 2023-12-02T03:18:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 35 33 36 b8 9f |..........4536..| 2023-12-02T03:18:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 18 9c f3 10 0a 00 00 00 01 01 04 00 00 00 |je..............| 2023-12-02T03:18:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000020 00 00 00 00 |....| 2023-12-02T03:18:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000024 2023-12-02T03:18:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:18:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:image_map_9333a204-121f-424e-849b-e6aec78f4224 2023-12-02T03:18:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:value (36 bytes) : 2023-12-02T03:18:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 35 33 36 38 9f |..........45368.| 2023-12-02T03:18:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:00000010 6a 65 79 3c 47 25 0a 00 00 00 01 01 04 00 00 00 |jey&)+0x10) [0x7f385e8d5500] 2023-12-02T03:23:49.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 5: rbd-mirror(+0x326f1b) [0x56182d6d0f1b] 2023-12-02T03:23:49.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0x103) [0x56182d6dbb03] 2023-12-02T03:23:49.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x56182d6dbe60] 2023-12-02T03:23:49.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 8: rbd-mirror(+0x91c588) [0x56182dcc6588] 2023-12-02T03:23:49.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 9: rbd-mirror(+0x91c570) [0x56182dcc6570] 2023-12-02T03:23:49.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x56182d6cb185] 2023-12-02T03:23:49.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 11: main() 2023-12-02T03:23:49.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 12: /lib64/libc.so.6(+0x3feb0) [0x7f385e43feb0] 2023-12-02T03:23:49.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 13: __libc_start_main() 2023-12-02T03:23:49.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 14: _start() 2023-12-02T03:23:49.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr:2023-12-02T03:23:49.202+0000 7f385e615640 -1 *** Caught signal (Aborted) ** 2023-12-02T03:23:49.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: in thread 7f385e615640 thread_name:rbd-mirror 2023-12-02T03:23:49.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 2023-12-02T03:23:49.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev) 2023-12-02T03:23:49.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f385e454db0] 2023-12-02T03:23:49.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f385e49c39a] 2023-12-02T03:23:49.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 3: pthread_cond_wait() 2023-12-02T03:23:49.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f385e8d5500] 2023-12-02T03:23:49.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 5: rbd-mirror(+0x326f1b) [0x56182d6d0f1b] 2023-12-02T03:23:49.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0x103) [0x56182d6dbb03] 2023-12-02T03:23:49.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x56182d6dbe60] 2023-12-02T03:23:49.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 8: rbd-mirror(+0x91c588) [0x56182dcc6588] 2023-12-02T03:23:49.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 9: rbd-mirror(+0x91c570) [0x56182dcc6570] 2023-12-02T03:23:49.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x56182d6cb185] 2023-12-02T03:23:49.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 11: main() 2023-12-02T03:23:49.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 12: /lib64/libc.so.6(+0x3feb0) [0x7f385e43feb0] 2023-12-02T03:23:49.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 13: __libc_start_main() 2023-12-02T03:23:49.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 14: _start() 2023-12-02T03:23:49.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-02T03:23:49.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 2023-12-02T03:23:49.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 0> 2023-12-02T03:23:49.202+0000 7f385e615640 -1 *** Caught signal (Aborted) ** 2023-12-02T03:23:49.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: in thread 7f385e615640 thread_name:rbd-mirror 2023-12-02T03:23:49.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 2023-12-02T03:23:49.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev) 2023-12-02T03:23:49.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f385e454db0] 2023-12-02T03:23:49.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f385e49c39a] 2023-12-02T03:23:49.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 3: pthread_cond_wait() 2023-12-02T03:23:49.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f385e8d5500] 2023-12-02T03:23:49.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 5: rbd-mirror(+0x326f1b) [0x56182d6d0f1b] 2023-12-02T03:23:49.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0x103) [0x56182d6dbb03] 2023-12-02T03:23:49.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x56182d6dbe60] 2023-12-02T03:23:49.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 8: rbd-mirror(+0x91c588) [0x56182dcc6588] 2023-12-02T03:23:49.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 9: rbd-mirror(+0x91c570) [0x56182dcc6570] 2023-12-02T03:23:49.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x56182d6cb185] 2023-12-02T03:23:49.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 11: main() 2023-12-02T03:23:49.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 12: /lib64/libc.so.6(+0x3feb0) [0x7f385e43feb0] 2023-12-02T03:23:49.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 13: __libc_start_main() 2023-12-02T03:23:49.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 14: _start() 2023-12-02T03:23:49.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-02T03:23:49.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 2023-12-02T03:23:49.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: -9999> 2023-12-02T03:23:49.202+0000 7f385e615640 -1 *** Caught signal (Aborted) ** 2023-12-02T03:23:49.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: in thread 7f385e615640 thread_name:rbd-mirror 2023-12-02T03:23:49.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 2023-12-02T03:23:49.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev) 2023-12-02T03:23:49.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f385e454db0] 2023-12-02T03:23:49.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f385e49c39a] 2023-12-02T03:23:49.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 3: pthread_cond_wait() 2023-12-02T03:23:49.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f385e8d5500] 2023-12-02T03:23:49.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 5: rbd-mirror(+0x326f1b) [0x56182d6d0f1b] 2023-12-02T03:23:49.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0x103) [0x56182d6dbb03] 2023-12-02T03:23:49.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x56182d6dbe60] 2023-12-02T03:23:49.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 8: rbd-mirror(+0x91c588) [0x56182dcc6588] 2023-12-02T03:23:49.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 9: rbd-mirror(+0x91c570) [0x56182dcc6570] 2023-12-02T03:23:49.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x56182d6cb185] 2023-12-02T03:23:49.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 11: main() 2023-12-02T03:23:49.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 12: /lib64/libc.so.6(+0x3feb0) [0x7f385e43feb0] 2023-12-02T03:23:49.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 13: __libc_start_main() 2023-12-02T03:23:49.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 14: _start() 2023-12-02T03:23:49.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-02T03:23:49.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr: 2023-12-02T03:23:49.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi057.stderr:daemon-helper: command crashed with signal 6 2023-12-02T03:23:51.510 INFO:tasks.daemonwatchdog.daemon_watchdog:thrasher.rbd_mirror.[cluster1] failed 2023-12-02T03:23:51.511 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-12-02T03:23:52.013 INFO:tasks.ceph.cluster1.osd.0:Sent signal 15 2023-12-02T03:23:52.014 INFO:tasks.ceph.cluster1.osd.1:Sent signal 15 2023-12-02T03:23:52.014 INFO:tasks.ceph.cluster1.osd.2:Sent signal 15 2023-12-02T03:23:52.014 INFO:tasks.ceph.cluster1.mon.a:Sent signal 15 2023-12-02T03:23:52.014 INFO:tasks.ceph.cluster1.mgr.x:Sent signal 15 2023-12-02T03:23:52.015 INFO:tasks.ceph.cluster1.osd.0.smithi029.stderr:2023-12-02T03:23:52.013+0000 7f5f38e75640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 (PID: 85303) UID: 0 2023-12-02T03:23:52.015 INFO:tasks.ceph.cluster1.osd.0.smithi029.stderr:2023-12-02T03:23:52.013+0000 7f5f38e75640 -1 osd.0 243 *** Got signal Terminated *** 2023-12-02T03:23:52.015 INFO:tasks.ceph.cluster1.osd.0.smithi029.stderr:2023-12-02T03:23:52.013+0000 7f5f38e75640 -1 osd.0 243 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-02T03:23:52.015 INFO:tasks.ceph.cluster1.osd.1.smithi029.stderr:2023-12-02T03:23:52.013+0000 7f5a75c46640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 (PID: 85301) UID: 0 2023-12-02T03:23:52.016 INFO:tasks.ceph.cluster1.osd.1.smithi029.stderr:2023-12-02T03:23:52.013+0000 7f5a75c46640 -1 osd.1 243 *** Got signal Terminated *** 2023-12-02T03:23:52.016 INFO:tasks.ceph.cluster1.osd.1.smithi029.stderr:2023-12-02T03:23:52.013+0000 7f5a75c46640 -1 osd.1 243 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-02T03:23:52.016 INFO:tasks.ceph.cluster1.osd.2.smithi029.stderr:2023-12-02T03:23:52.013+0000 7f13c37b1640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 (PID: 85305) UID: 0 2023-12-02T03:23:52.016 INFO:tasks.ceph.cluster1.osd.2.smithi029.stderr:2023-12-02T03:23:52.013+0000 7f13c37b1640 -1 osd.2 243 *** Got signal Terminated *** 2023-12-02T03:23:52.016 INFO:tasks.ceph.cluster1.osd.2.smithi029.stderr:2023-12-02T03:23:52.013+0000 7f13c37b1640 -1 osd.2 243 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-02T03:23:52.016 INFO:tasks.ceph.cluster1.mon.a.smithi029.stderr:2023-12-02T03:23:52.013+0000 7fb1f72e8640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster cluster1 -i a (PID: 84968) UID: 0 2023-12-02T03:23:52.017 INFO:tasks.ceph.cluster1.mon.a.smithi029.stderr:2023-12-02T03:23:52.013+0000 7fb1f72e8640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-02T03:23:52.017 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T03:23:52.014+0000 7f03e6f4a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster cluster1 -i x (PID: 84970) UID: 0 2023-12-02T03:23:52.017 INFO:tasks.ceph.cluster1.mgr.x.smithi029.stderr:2023-12-02T03:23:52.014+0000 7f03e6f4a640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-12-02T03:25:23.318 INFO:tasks.rbd_mirror_thrash:Failed to stop cluster2.client.mirror.0 2023-12-02T03:25:23.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 6 2023-12-02T03:25:23.319 ERROR:tasks.rbd_mirror_thrash:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/rbd_mirror_thrash.py", line 84, in _run self.do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/rbd_mirror_thrash.py", line 156, in do_thrash run.wait([daemon.proc], timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (101) after waiting for 600 seconds 2023-12-02T03:25:23.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr:*** Caught signal (Aborted) ** 2023-12-02T03:25:23.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: in thread 7f30eb926640 thread_name:rbd-mirror 2023-12-02T03:25:23.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev) 2023-12-02T03:25:23.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f30ebc54db0] 2023-12-02T03:25:23.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f30ebc9c39a] 2023-12-02T03:25:23.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 3: pthread_cond_wait() 2023-12-02T03:25:23.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f30ec0d5500] 2023-12-02T03:25:23.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 5: /lib64/librados.so.2(+0x13c6fb) [0x7f30ed9c16fb] 2023-12-02T03:25:23.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 6: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x55e64cd8daf8] 2023-12-02T03:25:23.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 7: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x55e64cd85d78] 2023-12-02T03:25:23.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 8: (rbd::mirror::Mirror::run()+0xb4) [0x55e64cd66424] 2023-12-02T03:25:23.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 9: main() 2023-12-02T03:25:23.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f30ebc3feb0] 2023-12-02T03:25:23.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 11: __libc_start_main() 2023-12-02T03:25:23.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 12: _start() 2023-12-02T03:25:23.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr:2023-12-02T03:25:23.319+0000 7f30eb926640 -1 *** Caught signal (Aborted) ** 2023-12-02T03:25:23.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: in thread 7f30eb926640 thread_name:rbd-mirror 2023-12-02T03:25:23.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 2023-12-02T03:25:23.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev) 2023-12-02T03:25:23.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f30ebc54db0] 2023-12-02T03:25:23.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f30ebc9c39a] 2023-12-02T03:25:23.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 3: pthread_cond_wait() 2023-12-02T03:25:23.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f30ec0d5500] 2023-12-02T03:25:23.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 5: /lib64/librados.so.2(+0x13c6fb) [0x7f30ed9c16fb] 2023-12-02T03:25:23.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 6: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x55e64cd8daf8] 2023-12-02T03:25:23.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 7: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x55e64cd85d78] 2023-12-02T03:25:23.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 8: (rbd::mirror::Mirror::run()+0xb4) [0x55e64cd66424] 2023-12-02T03:25:23.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 9: main() 2023-12-02T03:25:23.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f30ebc3feb0] 2023-12-02T03:25:23.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 11: __libc_start_main() 2023-12-02T03:25:23.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 12: _start() 2023-12-02T03:25:23.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-02T03:25:23.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 2023-12-02T03:25:23.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: -1923> 2023-12-02T03:15:18.172+0000 7f30e88ca640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 103486) UID: 1000 2023-12-02T03:25:23.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 0> 2023-12-02T03:25:23.319+0000 7f30eb926640 -1 *** Caught signal (Aborted) ** 2023-12-02T03:25:23.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: in thread 7f30eb926640 thread_name:rbd-mirror 2023-12-02T03:25:23.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 2023-12-02T03:25:23.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev) 2023-12-02T03:25:23.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f30ebc54db0] 2023-12-02T03:25:23.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f30ebc9c39a] 2023-12-02T03:25:23.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 3: pthread_cond_wait() 2023-12-02T03:25:23.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f30ec0d5500] 2023-12-02T03:25:23.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 5: /lib64/librados.so.2(+0x13c6fb) [0x7f30ed9c16fb] 2023-12-02T03:25:23.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 6: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x55e64cd8daf8] 2023-12-02T03:25:23.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 7: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x55e64cd85d78] 2023-12-02T03:25:23.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 8: (rbd::mirror::Mirror::run()+0xb4) [0x55e64cd66424] 2023-12-02T03:25:23.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 9: main() 2023-12-02T03:25:23.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f30ebc3feb0] 2023-12-02T03:25:23.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 11: __libc_start_main() 2023-12-02T03:25:23.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 12: _start() 2023-12-02T03:25:23.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-02T03:25:23.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 2023-12-02T03:25:23.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: -9999> 2023-12-02T03:15:18.172+0000 7f30e88ca640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 103486) UID: 1000 2023-12-02T03:25:23.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: -9998> 2023-12-02T03:25:23.319+0000 7f30eb926640 -1 *** Caught signal (Aborted) ** 2023-12-02T03:25:23.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: in thread 7f30eb926640 thread_name:rbd-mirror 2023-12-02T03:25:23.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 2023-12-02T03:25:23.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: ceph version 18.0.0-7631-gbed76814 (bed7681480ccb0192b91a35e691658867e1e96b7) reef (dev) 2023-12-02T03:25:23.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f30ebc54db0] 2023-12-02T03:25:23.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f30ebc9c39a] 2023-12-02T03:25:23.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 3: pthread_cond_wait() 2023-12-02T03:25:23.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f30ec0d5500] 2023-12-02T03:25:23.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 5: /lib64/librados.so.2(+0x13c6fb) [0x7f30ed9c16fb] 2023-12-02T03:25:23.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 6: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x55e64cd8daf8] 2023-12-02T03:25:23.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 7: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x55e64cd85d78] 2023-12-02T03:25:23.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 8: (rbd::mirror::Mirror::run()+0xb4) [0x55e64cd66424] 2023-12-02T03:25:23.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 9: main() 2023-12-02T03:25:23.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f30ebc3feb0] 2023-12-02T03:25:23.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 11: __libc_start_main() 2023-12-02T03:25:23.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 12: _start() 2023-12-02T03:25:23.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-02T03:25:23.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr: 2023-12-02T03:25:23.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi057.stderr:daemon-helper: command crashed with signal 6 2023-12-02T03:28:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:2023-12-02T03:28:25.107+0000 7f6ab6cd7640 0 monclient(hunting): authenticate timed out after 300 2023-12-02T03:28:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-02T03:28:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T03:28:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ ceph --cluster cluster2 service status 2023-12-02T03:33:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:2023-12-02T03:33:25.196+0000 7fced3097640 0 monclient(hunting): authenticate timed out after 300 2023-12-02T03:33:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-02T03:33:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:33:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:33:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image_pool in ${POOL} ${PARENT_POOL} 2023-12-02T03:33:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-12-02T03:33:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirror images 2023-12-02T03:33:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirror images' 2023-12-02T03:33:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror --namespace '' ls -l 2023-12-02T03:38:25.231 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: couldn't connect to the cluster! 2023-12-02T03:38:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-02T03:38:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:38:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:38:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirror mirror pool info 2023-12-02T03:38:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirror mirror pool info' 2023-12-02T03:38:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror --namespace '' mirror pool info 2023-12-02T03:43:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: couldn't connect to the cluster! 2023-12-02T03:43:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:43:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirror mirror pool status 2023-12-02T03:43:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:43:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirror mirror pool status' 2023-12-02T03:43:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T03:43:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror --namespace '' mirror pool status --verbose 2023-12-02T03:48:25.299 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: couldn't connect to the cluster! 2023-12-02T03:48:25.302 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T03:48:25.302 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T03:48:25.302 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ rbd --cluster cluster2 -p mirror --namespace '' ls 2023-12-02T03:53:25.335 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirror rbd_mirroring omap vals 2023-12-02T03:53:25.335 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirror rbd_mirroring omap vals' 2023-12-02T03:53:25.335 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rados --cluster cluster2 -p mirror --namespace '' listomapvals rbd_mirroring 2023-12-02T03:58:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-02T03:58:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirror rbd_mirror_leader omap vals 2023-12-02T03:58:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirror rbd_mirror_leader omap vals' 2023-12-02T03:58:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rados --cluster cluster2 -p mirror --namespace '' listomapvals rbd_mirror_leader 2023-12-02T04:03:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-02T04:03:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T04:03:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T04:03:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-12-02T04:03:25.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirror ns1 images 2023-12-02T04:03:25.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirror ns1 images' 2023-12-02T04:03:25.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 ls -l 2023-12-02T04:08:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: couldn't connect to the cluster! 2023-12-02T04:08:25.425 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-02T04:08:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T04:08:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirrorns1 mirror pool info 2023-12-02T04:08:25.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T04:08:25.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirrorns1 mirror pool info' 2023-12-02T04:08:25.427 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 mirror pool info 2023-12-02T04:13:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: couldn't connect to the cluster! 2023-12-02T04:13:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T04:13:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirrorns1 mirror pool status 2023-12-02T04:13:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T04:13:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirrorns1 mirror pool status' 2023-12-02T04:13:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T04:13:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 mirror pool status --verbose 2023-12-02T04:18:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: couldn't connect to the cluster! 2023-12-02T04:18:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T04:18:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T04:18:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns1 ls 2023-12-02T04:23:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirror ns1 rbd_mirroring omap vals 2023-12-02T04:23:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirror ns1 rbd_mirroring omap vals' 2023-12-02T04:23:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rados --cluster cluster2 -p mirror --namespace ns1 listomapvals rbd_mirroring 2023-12-02T04:28:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-02T04:28:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirror ns1 rbd_mirror_leader omap vals 2023-12-02T04:28:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirror ns1 rbd_mirror_leader omap vals' 2023-12-02T04:28:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rados --cluster cluster2 -p mirror --namespace ns1 listomapvals rbd_mirror_leader 2023-12-02T04:33:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-02T04:33:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T04:33:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T04:33:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-12-02T04:33:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirror ns2 images 2023-12-02T04:33:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirror ns2 images' 2023-12-02T04:33:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 ls -l 2023-12-02T04:38:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: couldn't connect to the cluster! 2023-12-02T04:38:25.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-02T04:38:25.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T04:38:25.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T04:38:25.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirrorns2 mirror pool info 2023-12-02T04:38:25.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirrorns2 mirror pool info' 2023-12-02T04:38:25.621 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 mirror pool info 2023-12-02T04:43:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: couldn't connect to the cluster! 2023-12-02T04:43:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T04:43:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirrorns2 mirror pool status 2023-12-02T04:43:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T04:43:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirrorns2 mirror pool status' 2023-12-02T04:43:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T04:43:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 mirror pool status --verbose 2023-12-02T04:48:25.687 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: couldn't connect to the cluster! 2023-12-02T04:48:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T04:48:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T04:48:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns2 ls 2023-12-02T04:53:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirror ns2 rbd_mirroring omap vals 2023-12-02T04:53:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirror ns2 rbd_mirroring omap vals' 2023-12-02T04:53:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rados --cluster cluster2 -p mirror --namespace ns2 listomapvals rbd_mirroring 2023-12-02T04:58:25.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-02T04:58:25.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirror ns2 rbd_mirror_leader omap vals 2023-12-02T04:58:25.751 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirror ns2 rbd_mirror_leader omap vals' 2023-12-02T04:58:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rados --cluster cluster2 -p mirror --namespace ns2 listomapvals rbd_mirror_leader 2023-12-02T05:03:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-02T05:03:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T05:03:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T05:03:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image_pool in ${POOL} ${PARENT_POOL} 2023-12-02T05:03:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-12-02T05:03:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirror_parent images 2023-12-02T05:03:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirror_parent images' 2023-12-02T05:03:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace '' ls -l 2023-12-02T05:08:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: couldn't connect to the cluster! 2023-12-02T05:08:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-02T05:08:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T05:08:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirror_parent mirror pool info 2023-12-02T05:08:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T05:08:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirror_parent mirror pool info' 2023-12-02T05:08:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace '' mirror pool info 2023-12-02T05:13:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: couldn't connect to the cluster! 2023-12-02T05:13:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T05:13:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T05:13:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirror_parent mirror pool status 2023-12-02T05:13:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirror_parent mirror pool status' 2023-12-02T05:13:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T05:13:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace '' mirror pool status --verbose 2023-12-02T05:18:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: couldn't connect to the cluster! 2023-12-02T05:18:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T05:18:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T05:18:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace '' ls 2023-12-02T05:23:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirror_parent rbd_mirroring omap vals 2023-12-02T05:23:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirror_parent rbd_mirroring omap vals' 2023-12-02T05:23:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace '' listomapvals rbd_mirroring 2023-12-02T05:28:25.945 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-02T05:28:25.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirror_parent rbd_mirror_leader omap vals 2023-12-02T05:28:25.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirror_parent rbd_mirror_leader omap vals' 2023-12-02T05:28:25.946 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace '' listomapvals rbd_mirror_leader 2023-12-02T05:33:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-02T05:33:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T05:33:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T05:33:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-12-02T05:33:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirror_parent ns1 images 2023-12-02T05:33:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirror_parent ns1 images' 2023-12-02T05:33:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 ls -l 2023-12-02T05:38:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: couldn't connect to the cluster! 2023-12-02T05:38:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-02T05:38:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T05:38:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T05:38:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirror_parentns1 mirror pool info 2023-12-02T05:38:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirror_parentns1 mirror pool info' 2023-12-02T05:38:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 mirror pool info 2023-12-02T05:43:26.043 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: couldn't connect to the cluster! 2023-12-02T05:43:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T05:43:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirror_parentns1 mirror pool status 2023-12-02T05:43:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T05:43:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirror_parentns1 mirror pool status' 2023-12-02T05:43:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T05:43:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 mirror pool status --verbose 2023-12-02T05:48:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: couldn't connect to the cluster! 2023-12-02T05:48:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T05:48:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T05:48:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace ns1 ls 2023-12-02T05:53:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirror_parent ns1 rbd_mirroring omap vals 2023-12-02T05:53:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirror_parent ns1 rbd_mirroring omap vals' 2023-12-02T05:53:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns1 listomapvals rbd_mirroring 2023-12-02T05:58:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-02T05:58:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirror_parent ns1 rbd_mirror_leader omap vals 2023-12-02T05:58:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirror_parent ns1 rbd_mirror_leader omap vals' 2023-12-02T05:58:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns1 listomapvals rbd_mirror_leader 2023-12-02T06:03:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-02T06:03:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T06:03:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T06:03:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-12-02T06:03:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirror_parent ns2 images' 2023-12-02T06:03:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirror_parent ns2 images 2023-12-02T06:03:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns2 ls -l 2023-12-02T06:08:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: couldn't connect to the cluster! 2023-12-02T06:08:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-02T06:08:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T06:08:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirror_parentns2 mirror pool info 2023-12-02T06:08:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T06:08:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirror_parentns2 mirror pool info' 2023-12-02T06:08:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns2 mirror pool info 2023-12-02T06:13:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: couldn't connect to the cluster! 2023-12-02T06:13:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T06:13:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T06:13:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirror_parentns2 mirror pool status' 2023-12-02T06:13:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirror_parentns2 mirror pool status 2023-12-02T06:13:26.242 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T06:13:26.242 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns2 mirror pool status --verbose 2023-12-02T06:18:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:rbd: couldn't connect to the cluster! 2023-12-02T06:18:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T06:18:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T06:18:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace ns2 ls 2023-12-02T06:23:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirror_parent ns2 rbd_mirroring omap vals 2023-12-02T06:23:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirror_parent ns2 rbd_mirroring omap vals' 2023-12-02T06:23:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns2 listomapvals rbd_mirroring 2023-12-02T06:28:26.337 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-02T06:28:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 mirror_parent ns2 rbd_mirror_leader omap vals 2023-12-02T06:28:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 mirror_parent ns2 rbd_mirror_leader omap vals' 2023-12-02T06:28:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns2 listomapvals rbd_mirror_leader 2023-12-02T06:33:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-02T06:33:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout: 2023-12-02T06:33:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 2023-12-02T06:33:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local ret 2023-12-02T06:33:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2023-12-02T06:33:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ seq 0 3 2023-12-02T06:33:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T06:33:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ daemon_pid_file cluster1:0 2023-12-02T06:33:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:0 2023-12-02T06:33:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T06:33:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-02T06:33:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:0 2023-12-02T06:33:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T06:33:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T06:33:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T06:33:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T06:33:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-02T06:33:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T06:33:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T06:33:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T06:33:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T06:33:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2023-12-02T06:33:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2023-12-02T06:33:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2023-12-02T06:33:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid ']' 2023-12-02T06:33:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster1 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid not exist)' 2023-12-02T06:33:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid not exist) 2023-12-02T06:33:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ continue 2023-12-02T06:33:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T06:33:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ daemon_pid_file cluster1:1 2023-12-02T06:33:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:1 2023-12-02T06:33:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T06:33:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-02T06:33:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:1 2023-12-02T06:33:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T06:33:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T06:33:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T06:33:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T06:33:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-02T06:33:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T06:33:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T06:33:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T06:33:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T06:33:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2023-12-02T06:33:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2023-12-02T06:33:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2023-12-02T06:33:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid ']' 2023-12-02T06:33:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster1 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid not exist)' 2023-12-02T06:33:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid not exist) 2023-12-02T06:33:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ continue 2023-12-02T06:33:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T06:33:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ daemon_pid_file cluster1:2 2023-12-02T06:33:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:2 2023-12-02T06:33:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T06:33:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-02T06:33:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:2 2023-12-02T06:33:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T06:33:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T06:33:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T06:33:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T06:33:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-02T06:33:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T06:33:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T06:33:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T06:33:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T06:33:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2023-12-02T06:33:26.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2023-12-02T06:33:26.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2023-12-02T06:33:26.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid ']' 2023-12-02T06:33:26.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster1 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid not exist)' 2023-12-02T06:33:26.435 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid not exist) 2023-12-02T06:33:26.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ continue 2023-12-02T06:33:26.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T06:33:26.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ daemon_pid_file cluster1:3 2023-12-02T06:33:26.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster1:3 2023-12-02T06:33:26.437 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T06:33:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-02T06:33:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster1:3 2023-12-02T06:33:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T06:33:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T06:33:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster1 2023-12-02T06:33:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T06:33:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-02T06:33:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster1 2023-12-02T06:33:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster1 2023-12-02T06:33:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T06:33:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T06:33:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2023-12-02T06:33:26.455 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2023-12-02T06:33:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2023-12-02T06:33:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid ']' 2023-12-02T06:33:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster1 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid not exist)' 2023-12-02T06:33:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid not exist) 2023-12-02T06:33:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ continue 2023-12-02T06:33:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2023-12-02T06:33:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ seq 0 3 2023-12-02T06:33:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T06:33:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ daemon_pid_file cluster2:0 2023-12-02T06:33:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster2:0 2023-12-02T06:33:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T06:33:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster2:0 cluster instance 2023-12-02T06:33:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster2:0 2023-12-02T06:33:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T06:33:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T06:33:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster2 2023-12-02T06:33:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=0 2023-12-02T06:33:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 0 = cluster2:0 ']' 2023-12-02T06:33:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster2 2023-12-02T06:33:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster2 2023-12-02T06:33:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=0 2023-12-02T06:33:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=0 2023-12-02T06:33:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2023-12-02T06:33:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2023-12-02T06:33:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2023-12-02T06:33:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid ']' 2023-12-02T06:33:26.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid not exist)' 2023-12-02T06:33:26.478 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid not exist) 2023-12-02T06:33:26.482 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ continue 2023-12-02T06:33:26.482 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T06:33:26.482 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ daemon_pid_file cluster2:1 2023-12-02T06:33:26.482 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster2:1 2023-12-02T06:33:26.482 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T06:33:26.482 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster2:1 cluster instance 2023-12-02T06:33:26.482 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster2:1 2023-12-02T06:33:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T06:33:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T06:33:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster2 2023-12-02T06:33:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=1 2023-12-02T06:33:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 1 = cluster2:1 ']' 2023-12-02T06:33:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster2 2023-12-02T06:33:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster2 2023-12-02T06:33:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=1 2023-12-02T06:33:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=1 2023-12-02T06:33:26.484 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2023-12-02T06:33:26.498 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2023-12-02T06:33:26.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2023-12-02T06:33:26.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid ']' 2023-12-02T06:33:26.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid not exist)' 2023-12-02T06:33:26.499 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid not exist) 2023-12-02T06:33:26.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ continue 2023-12-02T06:33:26.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T06:33:26.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ daemon_pid_file cluster2:2 2023-12-02T06:33:26.500 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster2:2 2023-12-02T06:33:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T06:33:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster2:2 cluster instance 2023-12-02T06:33:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster2:2 2023-12-02T06:33:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T06:33:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T06:33:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster2 2023-12-02T06:33:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=2 2023-12-02T06:33:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 2 = cluster2:2 ']' 2023-12-02T06:33:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster2 2023-12-02T06:33:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster2 2023-12-02T06:33:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=2 2023-12-02T06:33:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=2 2023-12-02T06:33:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'pid file' 2023-12-02T06:33:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2023-12-02T06:33:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2023-12-02T06:33:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid ']' 2023-12-02T06:33:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid not exist)' 2023-12-02T06:33:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid not exist) 2023-12-02T06:33:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ continue 2023-12-02T06:33:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T06:33:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ daemon_pid_file cluster2:3 2023-12-02T06:33:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster=cluster2:3 2023-12-02T06:33:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance 2023-12-02T06:33:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ set_cluster_instance cluster2:3 cluster instance 2023-12-02T06:33:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local val=cluster2:3 2023-12-02T06:33:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local cluster_var_name=cluster 2023-12-02T06:33:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ local instance_var_name=instance 2023-12-02T06:33:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ cluster=cluster2 2023-12-02T06:33:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ instance=3 2023-12-02T06:33:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ '[' 3 = cluster2:3 ']' 2023-12-02T06:33:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval cluster=cluster2 2023-12-02T06:33:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ cluster=cluster2 2023-12-02T06:33:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ eval instance=3 2023-12-02T06:33:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ instance=3 2023-12-02T06:33:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'pid file' 2023-12-02T06:33:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2023-12-02T06:33:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2023-12-02T06:33:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid ']' 2023-12-02T06:33:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo 'cluster2 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid not exist)' 2023-12-02T06:33:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid not exist) 2023-12-02T06:33:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ continue 2023-12-02T06:33:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 2023-12-02T06:33:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -z '' ']' 2023-12-02T06:33:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster instance 2023-12-02T06:33:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T06:33:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2023-12-02T06:38:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:2023-12-02T06:38:26.616+0000 7f350ab55640 0 monclient(hunting): authenticate timed out after 300 2023-12-02T06:38:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-02T06:38:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T06:38:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2023-12-02T06:43:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:2023-12-02T06:43:26.708+0000 7f3a40864640 0 monclient(hunting): authenticate timed out after 300 2023-12-02T06:43:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-02T06:43:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T06:43:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2023-12-02T06:48:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:2023-12-02T06:48:26.799+0000 7fb3f3221640 0 monclient(hunting): authenticate timed out after 300 2023-12-02T06:48:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-02T06:48:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ CEPH_ARGS= 2023-12-02T06:48:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2023-12-02T06:53:26.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:2023-12-02T06:53:26.891+0000 7fa728d83640 0 monclient(hunting): authenticate timed out after 300 2023-12-02T06:53:26.894 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-02T06:53:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2023-12-02T06:53:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stop_mirrors cluster1 2023-12-02T06:53:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1 2023-12-02T06:53:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local sig= 2023-12-02T06:53:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ seq 0 3 2023-12-02T06:53:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T06:53:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stop_mirror cluster1:0 '' 2023-12-02T06:53:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1:0 2023-12-02T06:53:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local sig= 2023-12-02T06:53:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test -n 1 2023-12-02T06:53:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 2023-12-02T06:53:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T06:53:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stop_mirror cluster1:1 '' 2023-12-02T06:53:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1:1 2023-12-02T06:53:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local sig= 2023-12-02T06:53:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test -n 1 2023-12-02T06:53:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 2023-12-02T06:53:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T06:53:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stop_mirror cluster1:2 '' 2023-12-02T06:53:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1:2 2023-12-02T06:53:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local sig= 2023-12-02T06:53:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test -n 1 2023-12-02T06:53:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 2023-12-02T06:53:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T06:53:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stop_mirror cluster1:3 '' 2023-12-02T06:53:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster1:3 2023-12-02T06:53:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local sig= 2023-12-02T06:53:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test -n 1 2023-12-02T06:53:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 2023-12-02T06:53:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2023-12-02T06:53:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stop_mirrors cluster2 2023-12-02T06:53:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2 2023-12-02T06:53:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local sig= 2023-12-02T06:53:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:++ seq 0 3 2023-12-02T06:53:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T06:53:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stop_mirror cluster2:0 '' 2023-12-02T06:53:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2:0 2023-12-02T06:53:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local sig= 2023-12-02T06:53:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test -n 1 2023-12-02T06:53:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 2023-12-02T06:53:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T06:53:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stop_mirror cluster2:1 '' 2023-12-02T06:53:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2:1 2023-12-02T06:53:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local sig= 2023-12-02T06:53:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test -n 1 2023-12-02T06:53:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 2023-12-02T06:53:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T06:53:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stop_mirror cluster2:2 '' 2023-12-02T06:53:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2:2 2023-12-02T06:53:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local sig= 2023-12-02T06:53:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test -n 1 2023-12-02T06:53:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 2023-12-02T06:53:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-02T06:53:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ stop_mirror cluster2:3 '' 2023-12-02T06:53:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local cluster=cluster2:3 2023-12-02T06:53:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ local sig= 2023-12-02T06:53:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test -n 1 2023-12-02T06:53:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ return 2023-12-02T06:53:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' -z 1 ']' 2023-12-02T06:53:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ test '' = /tmp/tmp.kebs98RNzE 2023-12-02T06:53:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ rm -Rf /tmp/tmp.kebs98RNzE 2023-12-02T06:53:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ '[' 110 -eq 0 ']' 2023-12-02T06:53:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ echo FAIL 2023-12-02T06:53:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stdout:FAIL 2023-12-02T06:53:26.905 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-12-02T06:53:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi057.stderr:+ exit 110 2023-12-02T06:53:26.906 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2023-12-02T06:53:26.906 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2023-12-02T06:53:27.237 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi057 with status 110: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bed7681480ccb0192b91a35e691658867e1e96b7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' 2023-12-02T06:53:27.327 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1533c33cce4b425c9df819b80ad0c7da Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi057 with status 110: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bed7681480ccb0192b91a35e691658867e1e96b7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' 2023-12-02T06:53:27.330 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2023-12-02T06:53:27.350 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2023-12-02T06:53:27.350 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror-thrash Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing 2023-12-02T06:53:27.365 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2023-12-02T06:53:27.390 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2023-12-02T06:53:27.391 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror-thrash Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/rbd_mirror_thrash.py", line 211, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing 2023-12-02T06:53:27.396 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-02T06:53:27.423 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2023-12-02T06:53:27.423 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T06:53:27.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi057.stderr:2023-12-02T06:53:27.422+0000 7f1a32eed640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 102632) UID: 1000 2023-12-02T06:58:26.489 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2023-12-02T06:58:26.547 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-02T06:58:26.558 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2023-12-02T06:58:26.558 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T06:58:26.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi057.stderr:2023-12-02T06:58:26.557+0000 7f4ed3c86640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 109655) UID: 1000 2023-12-02T07:03:25.629 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2023-12-02T07:03:25.635 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-02T07:03:25.648 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2023-12-02T07:03:25.648 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T07:03:25.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-02T07:03:25.651 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi057 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1' 2023-12-02T07:03:25.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Stopped 2023-12-02T07:03:25.654 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-02T07:03:25.696 ERROR:tasks.rbd_mirror.cluster2.client.mirror.0:tried to stop a non-running daemon 2023-12-02T07:03:25.699 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-02T07:03:25.747 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2023-12-02T07:03:25.747 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T07:03:25.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi057.stderr:2023-12-02T07:03:25.746+0000 7f7829588640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 104591) UID: 1000 2023-12-02T07:08:24.807 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2023-12-02T07:08:24.816 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-02T07:08:24.834 ERROR:tasks.rbd_mirror.cluster1.client.mirror.2:tried to stop a non-running daemon 2023-12-02T07:08:24.839 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-02T07:08:24.879 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2023-12-02T07:08:24.879 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T07:08:24.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi057.stderr:2023-12-02T07:08:24.877+0000 7fa849832640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 102090) UID: 1000 2023-12-02T07:13:23.950 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2023-12-02T07:13:23.956 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-02T07:13:23.969 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2023-12-02T07:13:23.969 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T07:13:23.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi057.stderr:2023-12-02T07:13:23.968+0000 7f5e7bf8f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 111516) UID: 1000 2023-12-02T07:18:23.042 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2023-12-02T07:18:23.049 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-02T07:18:23.062 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2023-12-02T07:18:23.062 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-02T07:20:23.110 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-02T07:20:23.111 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi057 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-12-02T07:20:23.129 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-02T07:20:23.129 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-02T07:20:23.129 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2023-12-02T07:20:23.130 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T07:20:23.133 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-12-02T07:20:23.133 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi057 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi057 with status 5: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0' 2023-12-02T07:20:23.149 INFO:tasks.ceph.cluster2.osd.0:Stopped 2023-12-02T07:20:23.149 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2023-12-02T07:20:23.149 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T07:20:23.150 INFO:tasks.ceph.cluster2.osd.1:Stopped 2023-12-02T07:20:23.150 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2023-12-02T07:20:23.150 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T07:20:23.150 INFO:tasks.ceph.cluster2.osd.2:Stopped 2023-12-02T07:20:23.151 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-02T07:20:23.151 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2023-12-02T07:20:23.151 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T07:20:23.151 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2023-12-02T07:20:23.152 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-02T07:20:23.152 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2023-12-02T07:20:23.152 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T07:20:23.152 INFO:tasks.ceph.cluster2.mon.a:Stopped 2023-12-02T07:20:23.153 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-02T07:20:23.153 DEBUG:teuthology.orchestra.run.smithi057:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-12-02T07:20:23.219 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi057.front.sepia.ceph.com 2023-12-02T07:20:23.220 DEBUG:teuthology.orchestra.run.smithi057:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2023-12-02T07:20:23.468 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi057.front.sepia.ceph.com 2023-12-02T07:20:23.468 DEBUG:teuthology.orchestra.run.smithi057:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2023-12-02T07:20:23.666 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi057.front.sepia.ceph.com 2023-12-02T07:20:23.667 DEBUG:teuthology.orchestra.run.smithi057:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2023-12-02T07:20:23.858 INFO:tasks.ceph:Archiving mon data... 2023-12-02T07:20:23.860 DEBUG:teuthology.misc:Transferring archived files from smithi057:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/yuriw-2023-12-01_18:01:25-rbd-main-distro-default-smithi/7474803/data/cluster2.mon.a.tgz 2023-12-02T07:20:23.860 DEBUG:teuthology.orchestra.run.smithi057:> mktemp 2023-12-02T07:20:23.880 INFO:teuthology.orchestra.run.smithi057.stdout:/tmp/tmp.f6ViafoXLL 2023-12-02T07:20:23.881 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.f6ViafoXLL 2023-12-02T07:20:24.068 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0666 /tmp/tmp.f6ViafoXLL 2023-12-02T07:20:24.175 DEBUG:teuthology.orchestra.remote:smithi057:/tmp/tmp.f6ViafoXLL is 438KB 2023-12-02T07:20:24.272 DEBUG:teuthology.orchestra.run.smithi057:> rm -fr /tmp/tmp.f6ViafoXLL 2023-12-02T07:20:24.287 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-02T07:20:24.288 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2023-12-02T07:20:24.291 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2023-12-02T07:20:24.367 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi057 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-12-02T07:20:24.382 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-02T07:20:24.395 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2023-12-02T07:20:24.396 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-02T07:22:24.441 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-02T07:22:24.442 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi029 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json' 2023-12-02T07:22:24.456 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-02T07:22:24.456 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-02T07:22:24.456 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2023-12-02T07:22:24.456 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T07:22:24.460 INFO:tasks.ceph.cluster1.osd.0:Stopped 2023-12-02T07:22:24.460 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2023-12-02T07:22:24.460 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T07:22:24.461 INFO:tasks.ceph.cluster1.osd.1:Stopped 2023-12-02T07:22:24.461 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2023-12-02T07:22:24.461 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T07:22:24.461 INFO:tasks.ceph.cluster1.osd.2:Stopped 2023-12-02T07:22:24.461 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-02T07:22:24.461 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2023-12-02T07:22:24.461 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T07:22:24.462 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2023-12-02T07:22:24.462 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-02T07:22:24.462 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2023-12-02T07:22:24.462 INFO:teuthology.orchestra.run:waiting for 300 2023-12-02T07:22:24.462 INFO:tasks.ceph.cluster1.mon.a:Stopped 2023-12-02T07:22:24.463 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-02T07:22:24.463 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-12-02T07:22:24.508 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi029.front.sepia.ceph.com 2023-12-02T07:22:24.508 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2023-12-02T07:22:24.636 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi029.front.sepia.ceph.com 2023-12-02T07:22:24.636 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2023-12-02T07:22:24.692 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi029.front.sepia.ceph.com 2023-12-02T07:22:24.693 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2023-12-02T07:22:24.769 INFO:tasks.ceph:Archiving mon data... 2023-12-02T07:22:24.769 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/yuriw-2023-12-01_18:01:25-rbd-main-distro-default-smithi/7474803/data/cluster1.mon.a.tgz 2023-12-02T07:22:24.769 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2023-12-02T07:22:24.785 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.Haz9lDS8Bu 2023-12-02T07:22:24.786 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.Haz9lDS8Bu 2023-12-02T07:22:24.931 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.Haz9lDS8Bu 2023-12-02T07:22:25.020 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.Haz9lDS8Bu is 267KB 2023-12-02T07:22:25.114 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.Haz9lDS8Bu 2023-12-02T07:22:25.129 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-02T07:22:25.129 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2023-12-02T07:22:25.171 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2023-12-02T07:22:25.254 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-02T07:22:25.272 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-02T07:22:25.272 INFO:tasks.ceph:Archiving crash dumps... 2023-12-02T07:22:25.274 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-12-01_18:01:25-rbd-main-distro-default-smithi/7474803/remote/smithi029/crash 2023-12-02T07:22:25.275 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-02T07:22:25.325 DEBUG:teuthology.misc:Transferring archived files from smithi057:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-12-01_18:01:25-rbd-main-distro-default-smithi/7474803/remote/smithi057/crash 2023-12-02T07:22:25.326 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-02T07:22:25.406 INFO:tasks.ceph:Compressing logs... 2023-12-02T07:22:25.406 DEBUG:teuthology.orchestra.run.smithi029:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-02T07:22:25.408 DEBUG:teuthology.orchestra.run.smithi057:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-02T07:22:25.439 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.88926.log 2023-12-02T07:22:25.439 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2023-12-02T07:22:25.439 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-client.admin.88926.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2023-12-02T07:22:25.439 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.88926.log.gz 2023-12-02T07:22:25.439 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84972.log 2023-12-02T07:22:25.439 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2023-12-02T07:22:25.440 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-client.admin.84972.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89091.log 2023-12-02T07:22:25.440 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.84972.log.gz 2023-12-02T07:22:25.440 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.88966.log 2023-12-02T07:22:25.440 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-client.admin.89091.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89091.log.gz 2023-12-02T07:22:25.440 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2023-12-02T07:22:25.440 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89052.log 2023-12-02T07:22:25.440 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-client.admin.88966.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.88966.log.gz 2023-12-02T07:22:25.440 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.88758.log 2023-12-02T07:22:25.451 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-client.admin.89052.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89262.log 2023-12-02T07:22:25.451 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89052.log.gz 2023-12-02T07:22:25.451 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-client.admin.88758.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.88674.log 2023-12-02T07:22:25.451 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.88758.log.gz 2023-12-02T07:22:25.451 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-client.admin.89262.log: gzip -5 --verbose -- /var/log/ceph/cluster1.log 2023-12-02T07:22:25.452 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89262.log.gz 2023-12-02T07:22:25.452 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-client.admin.88674.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.88839.log 2023-12-02T07:22:25.452 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.88674.log.gz 2023-12-02T07:22:25.452 INFO:teuthology.orchestra.run.smithi029.stderr: 92.9% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2023-12-02T07:22:25.452 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80493.log 2023-12-02T07:22:25.452 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.88838.log 2023-12-02T07:22:25.452 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-client.admin.88839.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.88839.log.gz 2023-12-02T07:22:25.452 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph.tmp-client.admin.80493.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.88450.log 2023-12-02T07:22:25.453 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.80493.log.gz 2023-12-02T07:22:25.453 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-client.admin.88838.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2023-12-02T07:22:25.453 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.88838.log.gz 2023-12-02T07:22:25.453 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.88534.log 2023-12-02T07:22:25.453 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-client.admin.88450.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.88450.log.gz 2023-12-02T07:22:25.453 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.88590.log 2023-12-02T07:22:25.453 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-client.admin.88534.log: gzip -5 0.0% --verbose -- /var/log/ceph/cluster1-osd.0.log 2023-12-02T07:22:25.453 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.88534.log.gz 2023-12-02T07:22:25.453 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-client.admin.88590.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89178.log 2023-12-02T07:22:25.454 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.88590.log.gz 2023-12-02T07:22:25.454 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.88716.log 2023-12-02T07:22:25.454 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-osd.0.log: /var/log/ceph/cluster1-client.admin.89178.log: gzip -5 --verbose -- 0.0% /var/log/ceph/cluster1-client.admin.85170.log 2023-12-02T07:22:25.454 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.89178.log.gz 2023-12-02T07:22:25.454 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-client.admin.88716.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.88716.log.gz 2023-12-02T07:22:25.454 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.88632.log 2023-12-02T07:22:25.454 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-client.admin.85170.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.88840.log 2023-12-02T07:22:25.454 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85170.log.gz 2023-12-02T07:22:25.454 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/cluster1.log.gz 2023-12-02T07:22:25.455 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-client.admin.88632.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.88632.log.gz 2023-12-02T07:22:25.455 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85371.log 2023-12-02T07:22:25.455 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-client.admin.88840.log: gzip -5 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.88840.log.gz 2023-12-02T07:22:25.455 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.90530.log 2023-12-02T07:22:25.455 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-client.admin.85371.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85371.log.gz 2023-12-02T07:22:25.455 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89220.log 2023-12-02T07:22:25.455 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-client.admin.90530.log: gzip -5 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90530.log.gz 2023-12-02T07:22:25.455 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.88965.log 2023-12-02T07:22:25.456 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-client.admin.89220.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89220.log.gz 2023-12-02T07:22:25.456 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85210.log 2023-12-02T07:22:25.456 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-client.admin.88965.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85309.log 2023-12-02T07:22:25.456 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.88965.log.gz 2023-12-02T07:22:25.456 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-client.admin.85210.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2023-12-02T07:22:25.456 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85210.log.gz 2023-12-02T07:22:25.456 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5/var/log/ceph/cluster1-client.admin.85309.log: --verbose -- /var/log/ceph/cluster1-client.admin.85130.log 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85309.log.gz 2023-12-02T07:22:25.457 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-12-02T07:22:25.457 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89092.log 2023-12-02T07:22:25.457 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-client.admin.85130.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85130.log.gz 2023-12-02T07:22:25.457 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cluster1-client.admin.89092.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89092.log.gz 2023-12-02T07:22:25.479 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116173.log 2023-12-02T07:22:25.479 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96794.log 2023-12-02T07:22:25.482 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116173.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107601.log 2023-12-02T07:22:25.482 INFO:teuthology.orchestra.run.smithi057.stderr: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116173.log.gz 2023-12-02T07:22:25.482 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.96794.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111865.log 2023-12-02T07:22:25.483 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.107601.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96300.log 2023-12-02T07:22:25.483 INFO:teuthology.orchestra.run.smithi057.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.96794.log.gz 2023-12-02T07:22:25.483 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.111865.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.107601.log.gz 2023-12-02T07:22:25.483 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94475.log 2023-12-02T07:22:25.483 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.111865.log.gz 2023-12-02T07:22:25.483 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.96300.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112565.log 2023-12-02T07:22:25.483 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.94475.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98061.log 2023-12-02T07:22:25.483 INFO:teuthology.orchestra.run.smithi057.stderr: 73.3% -- replaced with /var/log/ceph/cluster2-client.mirror.94475.log.gz 2023-12-02T07:22:25.484 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112565.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116194.log 2023-12-02T07:22:25.484 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.98061.log: gzip 87.3% -5 --verbose -- replaced with /var/log/ceph/cluster1-client.mirror.112565.log.gz 2023-12-02T07:22:25.484 INFO:teuthology.orchestra.run.smithi057.stderr: -- /var/log/ceph/cluster2-client.mirror.106500.log 2023-12-02T07:22:25.484 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116194.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95023.log 2023-12-02T07:22:25.484 INFO:teuthology.orchestra.run.smithi057.stderr: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.98061.log.gz 2023-12-02T07:22:25.484 INFO:teuthology.orchestra.run.smithi057.stderr: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116194.log.gz 2023-12-02T07:22:25.484 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.106500.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105923.log 2023-12-02T07:22:25.498 INFO:teuthology.orchestra.run.smithi057.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-client.mirror.96300.log.gz 2023-12-02T07:22:25.498 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.95023.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97525.log 2023-12-02T07:22:25.498 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.105923.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113005.log 2023-12-02T07:22:25.498 INFO:teuthology.orchestra.run.smithi057.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95023.log.gz 2023-12-02T07:22:25.498 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.97525.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.105923.log.gz 2023-12-02T07:22:25.498 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92324.log 2023-12-02T07:22:25.499 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113005.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.106500.log.gz 2023-12-02T07:22:25.499 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 88.4% --verbose -- -- replaced with /var/log/ceph/cluster1-client.mirror.97525.log.gz /var/log/ceph/cluster1-client.mirror.94393.log 2023-12-02T07:22:25.499 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-12-02T07:22:25.499 INFO:teuthology.orchestra.run.smithi057.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113005.log.gz 2023-12-02T07:22:25.499 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.92324.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115628.log 2023-12-02T07:22:25.499 INFO:teuthology.orchestra.run.smithi057.stderr: 53.4% -- replaced with /var/log/ceph/cluster1-client.mirror.92324.log.gz 2023-12-02T07:22:25.499 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.94393.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96936.log 2023-12-02T07:22:25.499 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115628.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94393.log.gz 2023-12-02T07:22:25.499 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100713.log 2023-12-02T07:22:25.500 INFO:teuthology.orchestra.run.smithi057.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115628.log.gz 2023-12-02T07:22:25.500 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.96936.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97126.log 2023-12-02T07:22:25.500 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100713.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115444.log 2023-12-02T07:22:25.500 INFO:teuthology.orchestra.run.smithi057.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.96936.log.gz 2023-12-02T07:22:25.500 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.97126.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.100713.log.gz 2023-12-02T07:22:25.500 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99306.log 2023-12-02T07:22:25.500 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115444.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.97126.log.gz 2023-12-02T07:22:25.500 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97254.log 2023-12-02T07:22:25.500 INFO:teuthology.orchestra.run.smithi057.stderr: 50.2% -- replaced with /var/log/ceph/cluster1-client.mirror.115444.log.gz 2023-12-02T07:22:25.501 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.99306.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94015.log 2023-12-02T07:22:25.501 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.97254.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100355.log 2023-12-02T07:22:25.501 INFO:teuthology.orchestra.run.smithi057.stderr:gzip/var/log/ceph/cluster1-client.mirror.94015.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105557.log 2023-12-02T07:22:25.501 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100355.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114545.log 2023-12-02T07:22:25.501 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.94015.log.gz 2023-12-02T07:22:25.501 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105557.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112845.log 2023-12-02T07:22:25.501 INFO:teuthology.orchestra.run.smithi057.stderr: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.105557.log.gz 2023-12-02T07:22:25.501 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114545.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115165.log 2023-12-02T07:22:25.501 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114545.log.gz 2023-12-02T07:22:25.502 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112845.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104714.log 2023-12-02T07:22:25.502 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.112845.log.gz 2023-12-02T07:22:25.502 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.115165.log: /var/log/ceph/cluster2-client.mirror.99822.log 2023-12-02T07:22:25.502 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104714.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98560.log 2023-12-02T07:22:25.502 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.115165.log.gz 2023-12-02T07:22:25.502 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99822.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.103503.log 2023-12-02T07:22:25.502 INFO:teuthology.orchestra.run.smithi057.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.104714.log.gz 2023-12-02T07:22:25.502 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94229.log 2023-12-02T07:22:25.503 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.98560.log: /var/log/ceph/cluster2-client.mirror.0.103503.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94148.log 2023-12-02T07:22:25.503 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.94229.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113745.log 2023-12-02T07:22:25.503 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.94148.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.94229.log.gz 2023-12-02T07:22:25.503 INFO:teuthology.orchestra.run.smithi057.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98560.log.gz 2023-12-02T07:22:25.503 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113745.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93923.log 2023-12-02T07:22:25.503 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.99822.log.gz 2023-12-02T07:22:25.503 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113745.log.gz 2023-12-02T07:22:25.503 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/cluster1-client.mirror.96437.log 2023-12-02T07:22:25.503 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.93923.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97147.log 2023-12-02T07:22:25.504 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.96437.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.93923.log.gz 2023-12-02T07:22:25.504 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.91879.log 2023-12-02T07:22:25.504 INFO:teuthology.orchestra.run.smithi057.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.96437.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115005.log 2023-12-02T07:22:25.504 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-12-02T07:22:25.504 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.97147.log: /var/log/ceph/cluster2-client.mirror.2.91879.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92290.log 2023-12-02T07:22:25.504 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115005.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116380.log 2023-12-02T07:22:25.504 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.92290.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.115005.log.gz 57.1% -- replaced with /var/log/ceph/cluster1-client.mirror.92290.log.gz 2023-12-02T07:22:25.504 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-12-02T07:22:25.504 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114645.log 2023-12-02T07:22:25.505 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116380.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114465.log 2023-12-02T07:22:25.505 INFO:teuthology.orchestra.run.smithi057.stderr: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116380.log.gz 2023-12-02T07:22:25.505 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114645.log: 87.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104173.log 2023-12-02T07:22:25.505 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.114645.log.gz 2023-12-02T07:22:25.505 INFO:teuthology.orchestra.run.smithi057.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99306.log.gz/var/log/ceph/cluster1-client.mirror.114465.log: 2023-12-02T07:22:25.505 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102649.log 2023-12-02T07:22:25.505 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114465.log.gz 2023-12-02T07:22:25.505 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104173.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116277.log 2023-12-02T07:22:25.505 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.102649.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109503.log 2023-12-02T07:22:25.506 INFO:teuthology.orchestra.run.smithi057.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104173.log.gz 2023-12-02T07:22:25.506 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116277.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116277.log.gz 2023-12-02T07:22:25.506 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116214.log 2023-12-02T07:22:25.506 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.109503.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102407.log 2023-12-02T07:22:25.524 INFO:teuthology.orchestra.run.smithi057.stderr: 93.4% 95.3% -- replaced with /var/log/ceph/cluster1-client.mirror.102649.log.gz 2023-12-02T07:22:25.524 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.97147.log.gz 2023-12-02T07:22:25.524 INFO:teuthology.orchestra.run.smithi057.stderr: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.2.91879.log.gz 2023-12-02T07:22:25.525 INFO:teuthology.orchestra.run.smithi057.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.109503.log.gz 2023-12-02T07:22:25.525 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116214.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112330.log 2023-12-02T07:22:25.525 INFO:teuthology.orchestra.run.smithi057.stderr: 6.1% -- replaced with /var/log/ceph/cluster2-client.mirror.116214.log.gz 2023-12-02T07:22:25.525 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101527.log 2023-12-02T07:22:25.525 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102407.log: /var/log/ceph/cluster1-client.mirror.112330.log: gzip 56.2% -5 --verbose -- /var/log/ceph/cluster2-client.admin.92700.log 2023-12-02T07:22:25.525 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.112330.log.gz 2023-12-02T07:22:25.525 INFO:teuthology.orchestra.run.smithi057.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102407.log.gz 2023-12-02T07:22:25.525 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101527.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106072.log 2023-12-02T07:22:25.526 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.92700.log: gzip -5 66.9% -- replaced with /var/log/ceph/cluster2-client.admin.92700.log.gz 2023-12-02T07:22:25.526 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.108559.log 2023-12-02T07:22:25.526 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.106072.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96191.log 2023-12-02T07:22:25.526 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106072.log.gz 2023-12-02T07:22:25.526 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.108559.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99902.log 2023-12-02T07:22:25.526 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.96191.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114145.log 2023-12-02T07:22:25.526 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99902.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.96191.log.gz 2023-12-02T07:22:25.526 INFO:teuthology.orchestra.run.smithi057.stderr: 92.5% 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.99902.log.gz 2023-12-02T07:22:25.527 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.108559.log.gz 2023-12-02T07:22:25.527 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95417.log 2023-12-02T07:22:25.527 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114145.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107622.log 2023-12-02T07:22:25.527 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.95417.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.114145.log.gz 2023-12-02T07:22:25.527 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115225.log 2023-12-02T07:22:25.527 INFO:teuthology.orchestra.run.smithi057.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.95417.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108830.log 2023-12-02T07:22:25.527 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-12-02T07:22:25.527 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.107622.log: /var/log/ceph/cluster1-client.mirror.115225.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85363.log 2023-12-02T07:22:25.528 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.108830.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.115225.log.gz 2023-12-02T07:22:25.528 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107256.log 2023-12-02T07:22:25.528 INFO:teuthology.orchestra.run.smithi057.stderr: 88.3%/var/log/ceph/cluster2-client.admin.85363.log: -- replaced with /var/log/ceph/cluster2-client.mirror.108830.log.gz 2023-12-02T07:22:25.528 INFO:teuthology.orchestra.run.smithi057.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85363.log.gz 2023-12-02T07:22:25.528 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98516.log 2023-12-02T07:22:25.528 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.107256.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.100392.log 2023-12-02T07:22:25.528 INFO:teuthology.orchestra.run.smithi057.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.107622.log.gz 2023-12-02T07:22:25.528 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.98516.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111183.log 2023-12-02T07:22:25.529 INFO:teuthology.orchestra.run.smithi057.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.107256.log.gz 2023-12-02T07:22:25.529 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.2.100392.log: gzip 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98516.log.gz 2023-12-02T07:22:25.529 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100182.log 2023-12-02T07:22:25.529 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.111183.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100042.log 2023-12-02T07:22:25.529 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.111183.log.gz 2023-12-02T07:22:25.529 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100182.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105783.log 2023-12-02T07:22:25.529 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100042.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100182.log.gz 2023-12-02T07:22:25.529 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112135.log 2023-12-02T07:22:25.529 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.100042.log.gz 2023-12-02T07:22:25.530 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105783.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103185.log 2023-12-02T07:22:25.530 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112135.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.105783.log.gz 2023-12-02T07:22:25.530 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115245.log 2023-12-02T07:22:25.530 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103185.log: 88.7% 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112135.log.gz 2023-12-02T07:22:25.530 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.103185.log.gz 2023-12-02T07:22:25.530 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104903.log 2023-12-02T07:22:25.530 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115245.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116390.log 2023-12-02T07:22:25.530 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104903.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.115245.log.gz 2023-12-02T07:22:25.531 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.104903.log.gz 2023-12-02T07:22:25.531 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94845.log 2023-12-02T07:22:25.531 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.116390.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91599.log 2023-12-02T07:22:25.531 INFO:teuthology.orchestra.run.smithi057.stderr: 9.2% -- replaced with /var/log/ceph/cluster2-client.admin.116390.log.gz 2023-12-02T07:22:25.531 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.94845.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107722.log 2023-12-02T07:22:25.531 INFO:teuthology.orchestra.run.smithi057.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.94845.log.gz 2023-12-02T07:22:25.531 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.91599.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.91599.log.gz 2023-12-02T07:22:25.531 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99349.log 2023-12-02T07:22:25.531 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.107722.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100271.log 2023-12-02T07:22:25.532 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.99349.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.107722.log.gz 2023-12-02T07:22:25.532 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103264.log 2023-12-02T07:22:25.532 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.100271.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.100271.log.gz 2023-12-02T07:22:25.532 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116306.log 2023-12-02T07:22:25.532 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103264.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103264.log.gz 2023-12-02T07:22:25.532 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114245.log 2023-12-02T07:22:25.532 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116306.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116306.log.gz 2023-12-02T07:22:25.532 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102342.log 2023-12-02T07:22:25.533 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114245.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114245.log.gz 2023-12-02T07:22:25.533 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98198.log 2023-12-02T07:22:25.533 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102342.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99349.log.gz 2023-12-02T07:22:25.533 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94981.log 2023-12-02T07:22:25.534 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.98198.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114045.log 2023-12-02T07:22:25.534 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.94981.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98198.log.gz 2023-12-02T07:22:25.535 INFO:teuthology.orchestra.run.smithi057.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.94981.log.gz 2023-12-02T07:22:25.536 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100587.log 2023-12-02T07:22:25.536 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114045.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114045.log.gz 2023-12-02T07:22:25.537 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95001.log 2023-12-02T07:22:25.542 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100587.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.2.100392.log.gz 2023-12-02T07:22:25.543 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113045.log 2023-12-02T07:22:25.548 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.95001.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.95001.log.gz 2023-12-02T07:22:25.549 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.112347.log 2023-12-02T07:22:25.549 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113045.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.113045.log.gz 2023-12-02T07:22:25.550 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92802.log 2023-12-02T07:22:25.561 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.112347.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.100587.log.gz 2023-12-02T07:22:25.562 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111928.log 2023-12-02T07:22:25.563 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.92802.log: 53.8% -- replaced with /var/log/ceph/cluster2-client.mirror.92802.log.gz 2023-12-02T07:22:25.563 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114505.log 2023-12-02T07:22:25.566 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.111928.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.112347.log.gz 2023-12-02T07:22:25.567 INFO:teuthology.orchestra.run.smithi057.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.111928.log.gz 2023-12-02T07:22:25.567 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112585.log 2023-12-02T07:22:25.567 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92410.log 2023-12-02T07:22:25.567 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.0.103503.log.gz 2023-12-02T07:22:25.567 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114505.log: /var/log/ceph/cluster1-client.mirror.112585.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.114505.log.gz 2023-12-02T07:22:25.567 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.112585.log.gz 2023-12-02T07:22:25.567 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116109.log 2023-12-02T07:22:25.568 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.92410.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.admin.92410.log.gz 2023-12-02T07:22:25.568 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98720.log 2023-12-02T07:22:25.568 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116109.log: 9.2%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94958.log 2023-12-02T07:22:25.568 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.116109.log.gz 2023-12-02T07:22:25.568 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.98720.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112525.log 2023-12-02T07:22:25.570 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101119.log 2023-12-02T07:22:25.570 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.94958.log: /var/log/ceph/cluster1-client.mirror.112525.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.112525.log.gz 2023-12-02T07:22:25.570 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99544.log 2023-12-02T07:22:25.571 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101119.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101119.log.gz 2023-12-02T07:22:25.571 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111906.log 2023-12-02T07:22:25.572 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99544.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.98720.log.gz 2023-12-02T07:22:25.573 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105214.log 2023-12-02T07:22:25.573 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.111906.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.94958.log.gz 2023-12-02T07:22:25.574 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114265.log 2023-12-02T07:22:25.574 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105214.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.105214.log.gz 2023-12-02T07:22:25.580 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105135.log 2023-12-02T07:22:25.580 INFO:teuthology.orchestra.run.smithi057.stderr: 92.2%/var/log/ceph/cluster1-client.mirror.114265.log: -- replaced with /var/log/ceph/cluster1-client.mirror.111906.log.gz 2023-12-02T07:22:25.580 INFO:teuthology.orchestra.run.smithi057.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.99544.log.gz 2023-12-02T07:22:25.580 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101348.log 2023-12-02T07:22:25.580 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.114265.log.gz 2023-12-02T07:22:25.580 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105135.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92878.log 2023-12-02T07:22:25.581 INFO:teuthology.orchestra.run.smithi057.stderr: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.105135.log.gz/var/log/ceph/cluster2-client.mirror.101348.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96119.log 2023-12-02T07:22:25.581 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-12-02T07:22:25.581 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115365.log 2023-12-02T07:22:25.581 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.92878.log: /var/log/ceph/cluster1-client.mirror.96119.log: 87.5% 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.96119.log.gz 2023-12-02T07:22:25.581 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.92878.log.gz 2023-12-02T07:22:25.581 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105231.log 2023-12-02T07:22:25.581 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115365.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112625.log 2023-12-02T07:22:25.581 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.115365.log.gz 2023-12-02T07:22:25.581 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105231.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104094.log 2023-12-02T07:22:25.582 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112625.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.105231.log.gz 2023-12-02T07:22:25.582 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.112625.log.gz 2023-12-02T07:22:25.582 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109978.log 2023-12-02T07:22:25.584 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104094.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104094.log.gz 2023-12-02T07:22:25.584 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.93743.log 2023-12-02T07:22:25.584 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97384.log 2023-12-02T07:22:25.584 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.109978.log: /var/log/ceph/cluster1-client.admin.93743.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109978.log.gz 2023-12-02T07:22:25.584 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97759.log 2023-12-02T07:22:25.584 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.93743.log.gz 2023-12-02T07:22:25.584 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.97384.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97384.log.gz 2023-12-02T07:22:25.584 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109785.log 2023-12-02T07:22:25.585 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113285.log 2023-12-02T07:22:25.585 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.97759.log: /var/log/ceph/cluster1-client.mirror.109785.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109785.log.gz 2023-12-02T07:22:25.590 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99501.log 2023-12-02T07:22:25.591 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113285.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.113285.log.gz 2023-12-02T07:22:25.592 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92307.log 2023-12-02T07:22:25.596 INFO:teuthology.orchestra.run.smithi029.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2023-12-02T07:22:25.597 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99501.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.99501.log.gz 2023-12-02T07:22:25.599 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95393.log 2023-12-02T07:22:25.599 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.92307.log: 56.4% -- replaced with /var/log/ceph/cluster1-client.mirror.92307.log.gz 2023-12-02T07:22:25.599 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99220.log 2023-12-02T07:22:25.600 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.95393.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.95393.log.gz 2023-12-02T07:22:25.600 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113705.log 2023-12-02T07:22:25.606 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.99220.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99220.log.gz 2023-12-02T07:22:25.606 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112745.log 2023-12-02T07:22:25.607 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113705.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113705.log.gz 2023-12-02T07:22:25.607 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115025.log 2023-12-02T07:22:25.608 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112745.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112745.log.gz 2023-12-02T07:22:25.609 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.100587.log 2023-12-02T07:22:25.610 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115025.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115025.log.gz 2023-12-02T07:22:25.610 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100498.log 2023-12-02T07:22:25.631 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.0.100587.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102283.log 2023-12-02T07:22:25.632 INFO:teuthology.orchestra.run.smithi029.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2023-12-02T07:22:25.633 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100498.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100498.log.gz 2023-12-02T07:22:25.646 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113465.log 2023-12-02T07:22:25.646 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102283.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102283.log.gz 2023-12-02T07:22:25.657 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105827.log 2023-12-02T07:22:25.657 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113465.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113465.log.gz 2023-12-02T07:22:25.670 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110870.log 2023-12-02T07:22:25.671 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.105827.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.105827.log.gz 2023-12-02T07:22:25.679 INFO:teuthology.orchestra.run.smithi057.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.0.100587.log.gz 2023-12-02T07:22:25.680 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98085.log 2023-12-02T07:22:25.681 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.110870.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.110870.log.gz 2023-12-02T07:22:25.682 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92683.log 2023-12-02T07:22:25.683 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.98085.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.98085.log.gz 2023-12-02T07:22:25.683 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97334.log 2023-12-02T07:22:25.684 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.92683.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.92683.log.gz 2023-12-02T07:22:25.685 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96602.log 2023-12-02T07:22:25.686 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.97334.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.97334.log.gz 2023-12-02T07:22:25.686 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106802.log 2023-12-02T07:22:25.699 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.96602.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112865.log 2023-12-02T07:22:25.700 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.106802.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.106802.log.gz 2023-12-02T07:22:25.713 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92785.log 2023-12-02T07:22:25.714 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112865.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.112865.log.gz 2023-12-02T07:22:25.715 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.95256.log 2023-12-02T07:22:25.715 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.92785.log: 56.4% -- replaced with /var/log/ceph/cluster2-client.mirror.92785.log.gz 2023-12-02T07:22:25.722 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105680.log 2023-12-02T07:22:25.728 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.0.95256.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.96602.log.gz 2023-12-02T07:22:25.735 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99329.log 2023-12-02T07:22:25.735 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.105680.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.105680.log.gz 2023-12-02T07:22:25.736 INFO:teuthology.orchestra.run.smithi057.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.95256.log.gz 2023-12-02T07:22:25.737 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108398.log 2023-12-02T07:22:25.742 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99329.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99329.log.gz 2023-12-02T07:22:25.743 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112685.log 2023-12-02T07:22:25.744 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.108398.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.108398.log.gz 2023-12-02T07:22:25.744 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98804.log 2023-12-02T07:22:25.745 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112685.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.112685.log.gz 2023-12-02T07:22:25.745 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114325.log 2023-12-02T07:22:25.746 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.98804.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.98804.log.gz 2023-12-02T07:22:25.747 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91847.log 2023-12-02T07:22:25.747 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114325.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.114325.log.gz 2023-12-02T07:22:25.748 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101289.log 2023-12-02T07:22:25.748 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.91847.log: 82.7% -- replaced with /var/log/ceph/cluster1-client.mirror.91847.log.gz 2023-12-02T07:22:25.749 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103019.log 2023-12-02T07:22:25.749 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101289.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.101289.log.gz 2023-12-02T07:22:25.749 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91179.log 2023-12-02T07:22:25.750 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103019.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103019.log.gz 2023-12-02T07:22:25.751 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96986.log 2023-12-02T07:22:25.751 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.91179.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.91179.log.gz 2023-12-02T07:22:25.752 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95862.log 2023-12-02T07:22:25.753 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.96986.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.96986.log.gz 2023-12-02T07:22:25.753 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112905.log 2023-12-02T07:22:25.754 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.95862.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.95862.log.gz 2023-12-02T07:22:25.754 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115555.log 2023-12-02T07:22:25.755 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112905.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.112905.log.gz 2023-12-02T07:22:25.755 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97786.log 2023-12-02T07:22:25.755 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115555.log: 48.6% -- replaced with /var/log/ceph/cluster1-client.mirror.115555.log.gz 2023-12-02T07:22:25.756 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107485.log 2023-12-02T07:22:25.757 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.97786.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.97786.log.gz 2023-12-02T07:22:25.757 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.93821.log 2023-12-02T07:22:25.758 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.107485.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.107485.log.gz 2023-12-02T07:22:25.758 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97006.log 2023-12-02T07:22:25.759 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.93821.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.93821.log.gz 2023-12-02T07:22:25.760 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109241.log 2023-12-02T07:22:25.760 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.97006.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.97006.log.gz 2023-12-02T07:22:25.761 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100248.log 2023-12-02T07:22:25.761 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.109241.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.109241.log.gz 2023-12-02T07:22:25.762 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115205.log 2023-12-02T07:22:25.762 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.100248.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.100248.log.gz 2023-12-02T07:22:25.763 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101102.log 2023-12-02T07:22:25.763 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115205.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.115205.log.gz 2023-12-02T07:22:25.764 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90969.log 2023-12-02T07:22:25.764 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101102.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101102.log.gz 2023-12-02T07:22:25.765 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96742.log 2023-12-02T07:22:25.765 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.90969.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90969.log.gz 2023-12-02T07:22:25.766 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103148.log 2023-12-02T07:22:25.766 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.96742.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.96742.log.gz 2023-12-02T07:22:25.767 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99759.log 2023-12-02T07:22:25.780 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103148.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95955.log 2023-12-02T07:22:25.787 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99759.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114105.log 2023-12-02T07:22:25.787 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.99759.log.gz 2023-12-02T07:22:25.800 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.95955.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100392.log 2023-12-02T07:22:25.801 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114105.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114105.log.gz 2023-12-02T07:22:25.802 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.97254.log.gz 2023-12-02T07:22:25.803 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115538.log 2023-12-02T07:22:25.805 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.100392.log: 94.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100392.log.gz 2023-12-02T07:22:25.806 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104215.log 2023-12-02T07:22:25.806 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115538.log: 68.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115538.log.gz 2023-12-02T07:22:25.806 INFO:teuthology.orchestra.run.smithi057.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95955.log.gz 2023-12-02T07:22:25.806 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101468.log 2023-12-02T07:22:25.820 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104215.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96880.log 2023-12-02T07:22:25.821 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101468.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101468.log.gz 2023-12-02T07:22:25.831 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108012.log 2023-12-02T07:22:25.831 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.96880.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.96880.log.gz 2023-12-02T07:22:25.844 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104821.log 2023-12-02T07:22:25.845 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.108012.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108012.log.gz 2023-12-02T07:22:25.855 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101865.log 2023-12-02T07:22:25.856 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104821.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104821.log.gz 2023-12-02T07:22:25.864 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109366.log 2023-12-02T07:22:25.877 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101865.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101865.log.gz 2023-12-02T07:22:25.880 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.115997.log 2023-12-02T07:22:25.881 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.109366.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.109366.log.gz 2023-12-02T07:22:25.881 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102342.log.gz 2023-12-02T07:22:25.883 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114525.log 2023-12-02T07:22:25.883 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.115997.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.admin.115997.log.gz 2023-12-02T07:22:25.883 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114305.log 2023-12-02T07:22:25.886 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114525.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114525.log.gz 2023-12-02T07:22:25.886 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104294.log 2023-12-02T07:22:25.886 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114305.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.114305.log.gz 2023-12-02T07:22:25.886 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106390.log 2023-12-02T07:22:25.913 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104294.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97078.log 2023-12-02T07:22:25.924 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.106390.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112203.log 2023-12-02T07:22:25.937 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.97078.log: 88.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104352.log 2023-12-02T07:22:25.937 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.112203.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.112203.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.97078.log.gz 2023-12-02T07:22:25.937 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-12-02T07:22:25.945 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107188.log 2023-12-02T07:22:25.956 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104352.log: -- replaced with /var/log/ceph/cluster2-client.mirror.106390.log.gz 2023-12-02T07:22:25.958 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.104352.log.gz 2023-12-02T07:22:25.968 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.112238.log 2023-12-02T07:22:25.971 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.107188.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107188.log.gz 2023-12-02T07:22:25.979 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104029.log 2023-12-02T07:22:25.980 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.112238.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.112238.log.gz 2023-12-02T07:22:25.983 INFO:teuthology.orchestra.run.smithi057.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.94148.log.gz 2023-12-02T07:22:25.983 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98368.log 2023-12-02T07:22:25.989 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104029.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.104029.log.gz 2023-12-02T07:22:25.989 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106325.log 2023-12-02T07:22:25.989 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.98368.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98368.log.gz 2023-12-02T07:22:25.989 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96413.log 2023-12-02T07:22:25.990 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.106325.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.106325.log.gz 2023-12-02T07:22:25.990 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110091.log 2023-12-02T07:22:25.990 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.96413.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.96413.log.gz 2023-12-02T07:22:25.990 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116101.log 2023-12-02T07:22:25.991 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.110091.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110091.log.gz 2023-12-02T07:22:25.993 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97737.log 2023-12-02T07:22:25.993 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116101.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116101.log.gz 2023-12-02T07:22:25.994 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95191.log 2023-12-02T07:22:25.994 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.97737.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.97737.log.gz 2023-12-02T07:22:25.994 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85675.log 2023-12-02T07:22:25.998 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.95191.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.95191.log.gz 2023-12-02T07:22:25.998 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92140.log 2023-12-02T07:22:25.998 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.85675.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85675.log.gz 2023-12-02T07:22:25.998 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95215.log 2023-12-02T07:22:25.998 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.92140.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92140.log.gz 2023-12-02T07:22:25.998 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.102871.log 2023-12-02T07:22:25.999 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.95215.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95215.log.gz 2023-12-02T07:22:25.999 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115265.log 2023-12-02T07:22:26.013 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.3.102871.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116064.log 2023-12-02T07:22:26.013 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115265.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115265.log.gz 2023-12-02T07:22:26.024 INFO:teuthology.orchestra.run.smithi057.stderr: 93.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109896.log 2023-12-02T07:22:26.024 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116064.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116064.log.gz 2023-12-02T07:22:26.025 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.3.102871.log.gz 2023-12-02T07:22:26.025 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109545.log 2023-12-02T07:22:26.027 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.109896.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.109896.log.gz 2023-12-02T07:22:26.027 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114685.log 2023-12-02T07:22:26.027 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.109545.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.109545.log.gz 2023-12-02T07:22:26.028 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103688.log 2023-12-02T07:22:26.028 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114685.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.114685.log.gz 2023-12-02T07:22:26.029 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116044.log 2023-12-02T07:22:26.029 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103688.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103688.log.gz 2023-12-02T07:22:26.031 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110697.log 2023-12-02T07:22:26.031 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.116044.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.admin.116044.log.gz 2023-12-02T07:22:26.031 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100313.log 2023-12-02T07:22:26.032 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.110697.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.110697.log.gz 2023-12-02T07:22:26.032 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112115.log 2023-12-02T07:22:26.034 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100313.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100313.log.gz 2023-12-02T07:22:26.034 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102738.log 2023-12-02T07:22:26.036 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.112115.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.112115.log.gz 2023-12-02T07:22:26.036 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92717.log 2023-12-02T07:22:26.040 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102738.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102738.log.gz 2023-12-02T07:22:26.040 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115607.log 2023-12-02T07:22:26.040 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101527.log.gz 2023-12-02T07:22:26.040 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.92717.log: 54.4% -- replaced with /var/log/ceph/cluster2-client.mirror.92717.log.gz 2023-12-02T07:22:26.041 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99478.log 2023-12-02T07:22:26.041 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115607.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100518.log 2023-12-02T07:22:26.041 INFO:teuthology.orchestra.run.smithi057.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115607.log.gz 2023-12-02T07:22:26.041 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.99478.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105081.log 2023-12-02T07:22:26.041 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100518.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100518.log.gz 2023-12-02T07:22:26.043 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107794.log 2023-12-02T07:22:26.043 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105081.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.105081.log.gz 2023-12-02T07:22:26.043 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111384.log 2023-12-02T07:22:26.043 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.107794.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.107794.log.gz 2023-12-02T07:22:26.045 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108241.log 2023-12-02T07:22:26.045 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.111384.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.111384.log.gz 2023-12-02T07:22:26.045 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114565.log 2023-12-02T07:22:26.045 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.108241.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.108241.log.gz 2023-12-02T07:22:26.049 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115666.log 2023-12-02T07:22:26.049 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114565.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99478.log.gz 2023-12-02T07:22:26.049 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.114565.log.gz 2023-12-02T07:22:26.049 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108767.log 2023-12-02T07:22:26.049 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.115666.log: -- /var/log/ceph/cluster2-client.mirror.99114.log 2023-12-02T07:22:26.049 INFO:teuthology.orchestra.run.smithi057.stderr: 55.8% -- replaced with /var/log/ceph/cluster1-client.mirror.115666.log.gz 2023-12-02T07:22:26.049 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.108767.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104432.log 2023-12-02T07:22:26.050 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99114.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.108767.log.gz 2023-12-02T07:22:26.050 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114585.log 2023-12-02T07:22:26.051 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104432.log: 79.3% -- replaced with /var/log/ceph/cluster2-client.mirror.104432.log.gz 2023-12-02T07:22:26.051 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100122.log 2023-12-02T07:22:26.051 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114585.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.114585.log.gz 2023-12-02T07:22:26.051 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99028.log 2023-12-02T07:22:26.052 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100122.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100122.log.gz 2023-12-02T07:22:26.052 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115701.log 2023-12-02T07:22:26.057 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99028.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99028.log.gz 2023-12-02T07:22:26.059 INFO:teuthology.orchestra.run.smithi057.stderr: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93991.log 2023-12-02T07:22:26.060 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.99114.log.gz 2023-12-02T07:22:26.060 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92819.log 2023-12-02T07:22:26.060 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115701.log: /var/log/ceph/cluster1-client.mirror.93991.log: 11.2% -- replaced with /var/log/ceph/cluster1-client.mirror.115701.log.gz 2023-12-02T07:22:26.060 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.93991.log.gz 2023-12-02T07:22:26.060 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91515.log 2023-12-02T07:22:26.064 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113145.log 2023-12-02T07:22:26.064 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.92819.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100355.log.gz/var/log/ceph/cluster2-client.admin.91515.log: 2023-12-02T07:22:26.064 INFO:teuthology.orchestra.run.smithi057.stderr: 54.2% -- replaced with /var/log/ceph/cluster2-client.mirror.92819.log.gz 2023-12-02T07:22:26.064 INFO:teuthology.orchestra.run.smithi057.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.91515.log.gz 2023-12-02T07:22:26.064 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105504.log 2023-12-02T07:22:26.064 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113145.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111360.log 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113145.log.gz 2023-12-02T07:22:26.065 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-12-02T07:22:26.065 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105504.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106120.log 2023-12-02T07:22:26.065 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.111360.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114165.log 2023-12-02T07:22:26.065 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.111360.log.gz 2023-12-02T07:22:26.065 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.106120.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.106120.log.gz 2023-12-02T07:22:26.068 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97842.log 2023-12-02T07:22:26.073 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.114165.log: -- /var/log/ceph/cluster1-client.mirror.113905.log 2023-12-02T07:22:26.074 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.114165.log.gz 2023-12-02T07:22:26.074 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.97842.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100481.log 2023-12-02T07:22:26.074 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113905.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.97842.log.gz 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.113905.log.gz 2023-12-02T07:22:26.074 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-12-02T07:22:26.074 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115045.log 2023-12-02T07:22:26.074 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100481.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104974.log 2023-12-02T07:22:26.074 INFO:teuthology.orchestra.run.smithi057.stderr: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100481.log.gz 2023-12-02T07:22:26.075 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115045.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.97880.log 2023-12-02T07:22:26.075 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104974.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.115045.log.gz 2023-12-02T07:22:26.075 INFO:teuthology.orchestra.run.smithi057.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104974.log.gz 2023-12-02T07:22:26.075 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115125.log 2023-12-02T07:22:26.075 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98656.log 2023-12-02T07:22:26.075 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115125.log: /var/log/ceph/cluster1-client.mirror.3.97880.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115125.log.gz 2023-12-02T07:22:26.076 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107393.log 2023-12-02T07:22:26.078 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.98656.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.98656.log.gz 2023-12-02T07:22:26.078 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97054.log 2023-12-02T07:22:26.078 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.107393.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.107393.log.gz 2023-12-02T07:22:26.078 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100800.log 2023-12-02T07:22:26.078 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.97054.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97054.log.gz 2023-12-02T07:22:26.080 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105152.log 2023-12-02T07:22:26.081 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100800.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.100800.log.gz 2023-12-02T07:22:26.081 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94369.log 2023-12-02T07:22:26.081 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105152.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.105152.log.gz 2023-12-02T07:22:26.083 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108149.log 2023-12-02T07:22:26.083 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.94369.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.94369.log.gz 2023-12-02T07:22:26.083 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85536.log 2023-12-02T07:22:26.085 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.108149.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.108149.log.gz 2023-12-02T07:22:26.085 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97190.log 2023-12-02T07:22:26.085 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.85536.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85536.log.gz 2023-12-02T07:22:26.085 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107031.log 2023-12-02T07:22:26.090 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.97190.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.97190.log.gz 2023-12-02T07:22:26.091 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111841.log 2023-12-02T07:22:26.091 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.107031.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.107031.log.gz 2023-12-02T07:22:26.092 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102070.log 2023-12-02T07:22:26.092 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.111841.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.111841.log.gz 2023-12-02T07:22:26.093 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92341.log 2023-12-02T07:22:26.100 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102070.log: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.3.97880.log.gz 2023-12-02T07:22:26.100 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114985.log 2023-12-02T07:22:26.101 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.92341.log: 53.1% -- replaced with /var/log/ceph/cluster1-client.mirror.92341.log.gz 2023-12-02T07:22:26.101 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103363.log 2023-12-02T07:22:26.102 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114985.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.114985.log.gz 2023-12-02T07:22:26.103 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113725.log 2023-12-02T07:22:26.103 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103363.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103363.log.gz 2023-12-02T07:22:26.104 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103851.log 2023-12-02T07:22:26.105 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113725.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.113725.log.gz 2023-12-02T07:22:26.105 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115888.log 2023-12-02T07:22:26.105 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103851.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103851.log.gz 2023-12-02T07:22:26.106 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99673.log 2023-12-02T07:22:26.106 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115888.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115888.log.gz 2023-12-02T07:22:26.107 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114945.log 2023-12-02T07:22:26.112 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99673.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99673.log.gz 2023-12-02T07:22:26.113 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.94475.log 2023-12-02T07:22:26.113 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114945.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.114945.log.gz 2023-12-02T07:22:26.113 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107702.log 2023-12-02T07:22:26.116 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.0.94475.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.94475.log.gz 2023-12-02T07:22:26.117 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114605.log 2023-12-02T07:22:26.117 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.107702.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.107702.log.gz 2023-12-02T07:22:26.118 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116120.log 2023-12-02T07:22:26.118 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114605.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114605.log.gz 2023-12-02T07:22:26.119 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107099.log 2023-12-02T07:22:26.119 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.116120.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.116120.log.gz 2023-12-02T07:22:26.120 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97816.log 2023-12-02T07:22:26.120 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.107099.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107099.log.gz 2023-12-02T07:22:26.121 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107881.log 2023-12-02T07:22:26.121 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.97816.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.97816.log.gz 2023-12-02T07:22:26.122 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104332.log 2023-12-02T07:22:26.123 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.107881.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.107881.log.gz 2023-12-02T07:22:26.123 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100735.log 2023-12-02T07:22:26.124 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104332.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104332.log.gz 2023-12-02T07:22:26.124 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.93769.log 2023-12-02T07:22:26.144 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100735.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98342.log 2023-12-02T07:22:26.145 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.93769.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.93769.log.gz 2023-12-02T07:22:26.146 INFO:teuthology.orchestra.run.smithi057.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.97759.log.gz 2023-12-02T07:22:26.147 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102775.log 2023-12-02T07:22:26.147 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.98342.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.98342.log.gz 2023-12-02T07:22:26.148 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113505.log 2023-12-02T07:22:26.150 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102775.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.102775.log.gz 2023-12-02T07:22:26.150 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94629.log 2023-12-02T07:22:26.151 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113505.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113505.log.gz 2023-12-02T07:22:26.151 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112465.log 2023-12-02T07:22:26.152 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.94629.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.94629.log.gz 2023-12-02T07:22:26.153 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.97626.log 2023-12-02T07:22:26.154 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112465.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112465.log.gz 2023-12-02T07:22:26.154 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.93472.log 2023-12-02T07:22:26.156 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.0.97626.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.0.97626.log.gz 2023-12-02T07:22:26.157 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109854.log 2023-12-02T07:22:26.165 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.1.93472.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103148.log.gz 2023-12-02T07:22:26.170 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114805.log 2023-12-02T07:22:26.173 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.109854.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.109854.log.gz 2023-12-02T07:22:26.173 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92841.log 2023-12-02T07:22:26.174 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114805.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114805.log.gz 2023-12-02T07:22:26.175 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99157.log 2023-12-02T07:22:26.175 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.92841.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.92841.log.gz 2023-12-02T07:22:26.176 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98740.log 2023-12-02T07:22:26.182 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99157.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.99157.log.gz 2023-12-02T07:22:26.182 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100062.log 2023-12-02T07:22:26.188 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.98740.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98740.log.gz 2023-12-02T07:22:26.188 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94104.log 2023-12-02T07:22:26.189 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100062.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.100062.log.gz 2023-12-02T07:22:26.190 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105426.log 2023-12-02T07:22:26.196 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.94104.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.94104.log.gz 2023-12-02T07:22:26.196 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103061.log 2023-12-02T07:22:26.198 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.105426.log: 95.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105426.log.gz 2023-12-02T07:22:26.198 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101564.log 2023-12-02T07:22:26.203 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103061.log: 94.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.93472.log.gz 2023-12-02T07:22:26.204 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91473.log 2023-12-02T07:22:26.207 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.101564.log: 94.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101564.log.gz 2023-12-02T07:22:26.207 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111473.log 2023-12-02T07:22:26.207 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.91473.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.91473.log.gz 2023-12-02T07:22:26.207 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96259.log 2023-12-02T07:22:26.209 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.111473.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111473.log.gz 2023-12-02T07:22:26.211 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93651.log 2023-12-02T07:22:26.211 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.96259.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.96259.log.gz 2023-12-02T07:22:26.211 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116031.log 2023-12-02T07:22:26.214 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.93651.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.93651.log.gz 2023-12-02T07:22:26.214 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111453.log 2023-12-02T07:22:26.214 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116031.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116031.log.gz 2023-12-02T07:22:26.214 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102491.log 2023-12-02T07:22:26.215 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.111453.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.111453.log.gz 2023-12-02T07:22:26.216 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95458.log 2023-12-02T07:22:26.217 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102491.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102491.log.gz 2023-12-02T07:22:26.218 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100994.log 2023-12-02T07:22:26.220 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.95458.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101348.log.gz 2023-12-02T07:22:26.224 INFO:teuthology.orchestra.run.smithi057.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.95458.log.gz 2023-12-02T07:22:26.224 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115961.log 2023-12-02T07:22:26.225 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.100994.log: /var/log/ceph/cluster2-client.mirror.104883.log 2023-12-02T07:22:26.226 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115961.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115961.log.gz 2023-12-02T07:22:26.226 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115427.log 2023-12-02T07:22:26.227 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104883.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104883.log.gz 2023-12-02T07:22:26.227 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101389.log 2023-12-02T07:22:26.228 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115427.log: 68.1% -- replaced with /var/log/ceph/cluster1-client.mirror.115427.log.gz 2023-12-02T07:22:26.229 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91673.log 2023-12-02T07:22:26.229 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101389.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101389.log.gz 2023-12-02T07:22:26.231 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103039.log 2023-12-02T07:22:26.231 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.91673.log: 23.1% -- replaced with /var/log/ceph/cluster2-client.mirror.91673.log.gz 2023-12-02T07:22:26.231 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106622.log 2023-12-02T07:22:26.233 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103039.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103039.log.gz 2023-12-02T07:22:26.233 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106981.log 2023-12-02T07:22:26.233 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.106622.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.106622.log.gz 2023-12-02T07:22:26.234 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108719.log 2023-12-02T07:22:26.240 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.106981.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.106981.log.gz 2023-12-02T07:22:26.240 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110962.log 2023-12-02T07:22:26.240 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.108719.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108719.log.gz 2023-12-02T07:22:26.240 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116237.log 2023-12-02T07:22:26.240 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.110962.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110962.log.gz 2023-12-02T07:22:26.240 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103343.log 2023-12-02T07:22:26.241 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116237.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116237.log.gz 2023-12-02T07:22:26.241 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114005.log 2023-12-02T07:22:26.243 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103343.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103343.log.gz 2023-12-02T07:22:26.243 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101740.log 2023-12-02T07:22:26.243 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114005.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114005.log.gz 2023-12-02T07:22:26.243 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115385.log 2023-12-02T07:22:26.252 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101740.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111769.log 2023-12-02T07:22:26.257 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115385.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115385.log.gz 2023-12-02T07:22:26.260 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94438.log 2023-12-02T07:22:26.260 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.111769.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.111769.log.gz 2023-12-02T07:22:26.267 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109631.log 2023-12-02T07:22:26.281 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.94438.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109217.log 2023-12-02T07:22:26.286 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.109631.log: 87.3% 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.94438.log.gz 2023-12-02T07:22:26.286 INFO:teuthology.orchestra.run.smithi029.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2023-12-02T07:22:26.294 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.109631.log.gz 2023-12-02T07:22:26.295 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105986.log 2023-12-02T07:22:26.295 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.109217.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.109217.log.gz 2023-12-02T07:22:26.304 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103609.log 2023-12-02T07:22:26.313 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105986.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108490.log 2023-12-02T07:22:26.314 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103609.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103609.log.gz 2023-12-02T07:22:26.317 INFO:teuthology.orchestra.run.smithi057.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.105986.log.gz 2023-12-02T07:22:26.334 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108808.log 2023-12-02T07:22:26.334 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.108490.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108490.log.gz 2023-12-02T07:22:26.345 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90630.log 2023-12-02T07:22:26.347 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.108808.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108808.log.gz 2023-12-02T07:22:26.354 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94587.log 2023-12-02T07:22:26.364 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.90630.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109193.log 2023-12-02T07:22:26.367 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.94587.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90630.log.gz 2023-12-02T07:22:26.369 INFO:teuthology.orchestra.run.smithi057.stderr: 93.4% 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104215.log.gz 2023-12-02T07:22:26.370 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.94587.log.gz 2023-12-02T07:22:26.371 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98241.log 2023-12-02T07:22:26.371 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.109193.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109193.log.gz 2023-12-02T07:22:26.372 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114825.log 2023-12-02T07:22:26.376 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.98241.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98241.log.gz 2023-12-02T07:22:26.378 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100540.log 2023-12-02T07:22:26.378 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114825.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114825.log.gz 2023-12-02T07:22:26.379 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99779.log 2023-12-02T07:22:26.398 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100540.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105352.log 2023-12-02T07:22:26.403 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.99779.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.99779.log.gz 2023-12-02T07:22:26.416 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115850.log 2023-12-02T07:22:26.429 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105352.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113665.log 2023-12-02T07:22:26.429 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115850.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115850.log.gz 2023-12-02T07:22:26.437 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.91656.log 2023-12-02T07:22:26.438 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113665.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.113665.log.gz 2023-12-02T07:22:26.447 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102449.log 2023-12-02T07:22:26.456 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.0.91656.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100952.log 2023-12-02T07:22:26.459 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102449.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.91656.log.gz 2023-12-02T07:22:26.465 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94173.log 2023-12-02T07:22:26.466 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100952.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100952.log.gz 2023-12-02T07:22:26.474 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106207.log 2023-12-02T07:22:26.482 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.94173.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104294.log.gz 2023-12-02T07:22:26.483 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.94173.log.gz 2023-12-02T07:22:26.491 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103061.log.gz 2023-12-02T07:22:26.492 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91431.log 2023-12-02T07:22:26.494 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.106207.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.106207.log.gz 2023-12-02T07:22:26.494 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99962.log 2023-12-02T07:22:26.495 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.91431.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.91431.log.gz 2023-12-02T07:22:26.495 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113645.log 2023-12-02T07:22:26.496 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99962.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.99962.log.gz 2023-12-02T07:22:26.497 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.111533.log 2023-12-02T07:22:26.498 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113645.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.113645.log.gz 2023-12-02T07:22:26.498 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95501.log 2023-12-02T07:22:26.512 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.0.111533.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109169.log 2023-12-02T07:22:26.522 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.95501.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114865.log 2023-12-02T07:22:26.522 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.95501.log.gz 2023-12-02T07:22:26.522 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.109169.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109169.log.gz 2023-12-02T07:22:26.528 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113325.log 2023-12-02T07:22:26.529 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114865.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114865.log.gz 2023-12-02T07:22:26.541 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112445.log 2023-12-02T07:22:26.541 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113325.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.113325.log.gz 2023-12-02T07:22:26.549 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100994.log.gz 2023-12-02T07:22:26.551 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102070.log.gz 2023-12-02T07:22:26.552 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.109672.log 2023-12-02T07:22:26.552 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112445.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112445.log.gz 2023-12-02T07:22:26.553 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112765.log 2023-12-02T07:22:26.562 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.2.109672.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95886.log 2023-12-02T07:22:26.572 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112765.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.112765.log.gz 2023-12-02T07:22:26.574 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.101564.log 2023-12-02T07:22:26.575 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.95886.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.95886.log.gz 2023-12-02T07:22:26.587 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101777.log 2023-12-02T07:22:26.596 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.0.101564.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97681.log 2023-12-02T07:22:26.597 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101777.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100735.log.gz 2023-12-02T07:22:26.599 INFO:teuthology.orchestra.run.smithi057.stderr: 88.7% 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.101564.log.gz 2023-12-02T07:22:26.599 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116339.log 2023-12-02T07:22:26.606 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.97681.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.97681.log.gz 2023-12-02T07:22:26.606 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97546.log 2023-12-02T07:22:26.606 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116339.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.mirror.116339.log.gz 2023-12-02T07:22:26.606 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102960.log 2023-12-02T07:22:26.608 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.97546.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101777.log.gz 2023-12-02T07:22:26.608 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.2.109672.log.gz 2023-12-02T07:22:26.611 INFO:teuthology.orchestra.run.smithi057.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.97546.log.gz 2023-12-02T07:22:26.616 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110890.log 2023-12-02T07:22:26.616 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102960.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99922.log 2023-12-02T07:22:26.616 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.110890.log: 86.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102960.log.gz 2023-12-02T07:22:26.616 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110890.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2023-12-02T07:22:26.616 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-12-02T07:22:26.616 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99922.log: gzip -5 --verbose -- 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.99922.log.gz 2023-12-02T07:22:26.616 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/cluster1-client.mirror.112645.log 2023-12-02T07:22:26.616 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97030.log 2023-12-02T07:22:26.617 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-osd.1.log: /var/log/ceph/cluster1-client.mirror.112645.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.112645.log.gz 2023-12-02T07:22:26.617 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111336.log 2023-12-02T07:22:26.617 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.97030.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106939.log 2023-12-02T07:22:26.617 INFO:teuthology.orchestra.run.smithi057.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.97030.log.gz 2023-12-02T07:22:26.622 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.111336.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97308.log 2023-12-02T07:22:26.623 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.106939.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.111336.log.gz 2023-12-02T07:22:26.623 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113365.log 2023-12-02T07:22:26.623 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.97308.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.97308.log.gz 2023-12-02T07:22:26.623 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114365.log 2023-12-02T07:22:26.623 INFO:teuthology.orchestra.run.smithi057.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.106939.log.gz 2023-12-02T07:22:26.623 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113365.log: gzip 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.113365.log.gz 2023-12-02T07:22:26.623 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103787.log 2023-12-02T07:22:26.623 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94039.log 2023-12-02T07:22:26.624 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103787.log: /var/log/ceph/cluster1-client.mirror.114365.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103787.log.gz 87.5% 2023-12-02T07:22:26.624 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.114365.log.gz 2023-12-02T07:22:26.624 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96565.log 2023-12-02T07:22:26.624 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99862.log 2023-12-02T07:22:26.624 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.96565.log: /var/log/ceph/cluster1-client.mirror.94039.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.94039.log.gz 2023-12-02T07:22:26.624 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91137.log 2023-12-02T07:22:26.628 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99862.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.99862.log.gz 2023-12-02T07:22:26.628 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94797.log 2023-12-02T07:22:26.628 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.91137.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.91137.log.gz 2023-12-02T07:22:26.628 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90252.log 2023-12-02T07:22:26.628 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.94797.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.94797.log.gz 2023-12-02T07:22:26.629 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106301.log 2023-12-02T07:22:26.629 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.90252.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90252.log.gz 2023-12-02T07:22:26.629 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105851.log 2023-12-02T07:22:26.631 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.106301.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.106301.log.gz 2023-12-02T07:22:26.631 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107988.log 2023-12-02T07:22:26.631 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.105851.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.105851.log.gz 2023-12-02T07:22:26.631 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103767.log 2023-12-02T07:22:26.632 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.107988.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.107988.log.gz 2023-12-02T07:22:26.632 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95099.log 2023-12-02T07:22:26.634 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103767.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103767.log.gz 2023-12-02T07:22:26.634 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107413.log 2023-12-02T07:22:26.634 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.95099.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.95099.log.gz 2023-12-02T07:22:26.635 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115683.log 2023-12-02T07:22:26.639 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.107413.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107413.log.gz 2023-12-02T07:22:26.639 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101949.log 2023-12-02T07:22:26.640 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.115683.log: 50.8% -- replaced with /var/log/ceph/cluster1-client.admin.115683.log.gz 2023-12-02T07:22:26.640 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103002.log 2023-12-02T07:22:26.641 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101949.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101949.log.gz 2023-12-02T07:22:26.641 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108466.log 2023-12-02T07:22:26.642 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103002.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103002.log.gz 2023-12-02T07:22:26.642 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113565.log 2023-12-02T07:22:26.644 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.108466.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.108466.log.gz 2023-12-02T07:22:26.644 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102028.log 2023-12-02T07:22:26.644 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113565.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113565.log.gz 2023-12-02T07:22:26.645 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116009.log 2023-12-02T07:22:26.647 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102028.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102028.log.gz 2023-12-02T07:22:26.647 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113225.log 2023-12-02T07:22:26.651 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.116009.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.admin.116009.log.gz 2023-12-02T07:22:26.652 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111316.log 2023-12-02T07:22:26.652 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113225.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.113225.log.gz 2023-12-02T07:22:26.652 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103306.log 2023-12-02T07:22:26.652 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.111316.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.111316.log.gz 2023-12-02T07:22:26.652 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100676.log 2023-12-02T07:22:26.660 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103306.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110428.log 2023-12-02T07:22:26.673 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100676.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105061.log 2023-12-02T07:22:26.674 INFO:teuthology.orchestra.run.smithi057.stderr: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100676.log.gz 2023-12-02T07:22:26.674 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.110428.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110428.log.gz 2023-12-02T07:22:26.688 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114905.log 2023-12-02T07:22:26.689 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105061.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.105061.log.gz 2023-12-02T07:22:26.707 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111091.log 2023-12-02T07:22:26.707 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114905.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114905.log.gz 2023-12-02T07:22:26.715 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98536.log 2023-12-02T07:22:26.716 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.111091.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.111091.log.gz 2023-12-02T07:22:26.732 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98962.log 2023-12-02T07:22:26.732 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.98536.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.98536.log.gz 2023-12-02T07:22:26.744 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101448.log 2023-12-02T07:22:26.749 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.98962.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98962.log.gz 2023-12-02T07:22:26.753 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108940.log 2023-12-02T07:22:26.754 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101448.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.105352.log.gz 2023-12-02T07:22:26.767 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104608.log 2023-12-02T07:22:26.777 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.108940.log: 87.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103407.log 2023-12-02T07:22:26.779 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104608.log: -- replaced with /var/log/ceph/cluster1-client.mirror.108940.log.gz 2023-12-02T07:22:26.789 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98312.log 2023-12-02T07:22:26.789 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103407.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103407.log.gz 2023-12-02T07:22:26.798 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116152.log 2023-12-02T07:22:26.798 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.98312.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.98312.log.gz 2023-12-02T07:22:26.812 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109145.log 2023-12-02T07:22:26.812 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116152.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.mirror.116152.log.gz 2023-12-02T07:22:26.821 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91009.log 2023-12-02T07:22:26.821 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.109145.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.109145.log.gz 2023-12-02T07:22:26.830 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110071.log 2023-12-02T07:22:26.843 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.91009.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.91009.log.gz 2023-12-02T07:22:26.845 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.106538.log 2023-12-02T07:22:26.847 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.110071.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.110071.log.gz 2023-12-02T07:22:26.855 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112022.log 2023-12-02T07:22:26.867 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.1.106538.log: -- replaced with /var/log/ceph/cluster2-client.mirror.104608.log.gz 2023-12-02T07:22:26.878 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116257.log 2023-12-02T07:22:26.879 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112022.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112022.log.gz 2023-12-02T07:22:26.887 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115065.log 2023-12-02T07:22:26.887 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.116257.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.1.106538.log.gz 2023-12-02T07:22:26.892 INFO:teuthology.orchestra.run.smithi057.stderr: 9.2% -- replaced with /var/log/ceph/cluster2-client.admin.116257.log.gz 2023-12-02T07:22:26.904 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2023-12-02T07:22:26.906 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115065.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101740.log.gz 2023-12-02T07:22:26.907 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.105504.log.gz 2023-12-02T07:22:26.907 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96521.log 2023-12-02T07:22:26.907 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.115065.log.gz 2023-12-02T07:22:26.907 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102390.log 2023-12-02T07:22:26.907 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106144.log 2023-12-02T07:22:26.908 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102390.log: /var/log/ceph/cluster1-client.mirror.96521.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.102390.log.gz 2023-12-02T07:22:26.912 INFO:teuthology.orchestra.run.smithi057.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.96521.log.gz 2023-12-02T07:22:26.912 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110384.log 2023-12-02T07:22:26.913 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.106144.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.106144.log.gz 2023-12-02T07:22:26.913 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91815.log 2023-12-02T07:22:26.914 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.110384.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.110384.log.gz 2023-12-02T07:22:26.916 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96211.log 2023-12-02T07:22:26.916 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.91815.log: 91.1% -- replaced with /var/log/ceph/cluster1-client.mirror.91815.log.gz 2023-12-02T07:22:26.917 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95748.log 2023-12-02T07:22:26.917 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.96211.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.96211.log.gz 2023-12-02T07:22:26.918 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105638.log 2023-12-02T07:22:26.919 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.95748.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95748.log.gz 2023-12-02T07:22:26.920 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114665.log 2023-12-02T07:22:26.921 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105638.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.105638.log.gz 2023-12-02T07:22:26.921 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114085.log 2023-12-02T07:22:26.922 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114665.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114665.log.gz 2023-12-02T07:22:26.922 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91221.log 2023-12-02T07:22:26.924 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114085.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.114085.log.gz 2023-12-02T07:22:26.924 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110452.log 2023-12-02T07:22:26.924 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.91221.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.91221.log.gz 2023-12-02T07:22:26.925 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110565.log 2023-12-02T07:22:26.925 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.110452.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110452.log.gz 2023-12-02T07:22:26.926 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116267.log 2023-12-02T07:22:26.928 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.110565.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.110565.log.gz 2023-12-02T07:22:26.930 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110673.log 2023-12-02T07:22:26.930 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116267.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116267.log.gz 2023-12-02T07:22:26.930 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102233.log 2023-12-02T07:22:26.931 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.110673.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110673.log.gz 2023-12-02T07:22:26.932 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97452.log 2023-12-02T07:22:26.933 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102233.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102233.log.gz 2023-12-02T07:22:26.933 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.91919.log 2023-12-02T07:22:26.934 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.97452.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.97452.log.gz 2023-12-02T07:22:26.934 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101485.log 2023-12-02T07:22:26.935 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.3.91919.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111135.log 2023-12-02T07:22:26.939 INFO:teuthology.orchestra.run.smithi057.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.3.91919.log.gz 2023-12-02T07:22:26.939 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101485.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101485.log.gz 2023-12-02T07:22:26.949 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100203.log 2023-12-02T07:22:26.950 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.111135.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111135.log.gz 2023-12-02T07:22:26.951 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99435.log 2023-12-02T07:22:26.952 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100203.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.100203.log.gz 2023-12-02T07:22:26.952 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116185.log 2023-12-02T07:22:26.957 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.99435.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99435.log.gz 2023-12-02T07:22:26.959 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94297.log 2023-12-02T07:22:26.959 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.116185.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.116185.log.gz 2023-12-02T07:22:26.959 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.96300.log 2023-12-02T07:22:26.961 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.94297.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.94297.log.gz 2023-12-02T07:22:26.961 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85496.log 2023-12-02T07:22:26.975 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.1.96300.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108579.log 2023-12-02T07:22:26.975 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100540.log.gz 2023-12-02T07:22:26.975 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.85496.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85496.log.gz 2023-12-02T07:22:26.984 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96543.log 2023-12-02T07:22:26.986 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.108579.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108579.log.gz 2023-12-02T07:22:26.997 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103466.log 2023-12-02T07:22:26.998 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.96543.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.96543.log.gz 2023-12-02T07:22:27.011 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116164.log 2023-12-02T07:22:27.033 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103466.log: 93.3%gzip -5 --verbose -- /var/log/ceph/cluster2.log 2023-12-02T07:22:27.034 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116164.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116164.log.gz 2023-12-02T07:22:27.036 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102449.log.gz 2023-12-02T07:22:27.037 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.1.96300.log.gz 2023-12-02T07:22:27.037 INFO:teuthology.orchestra.run.smithi057.stderr: 96.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.111533.log.gz 2023-12-02T07:22:27.038 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105194.log 2023-12-02T07:22:27.038 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101698.log 2023-12-02T07:22:27.041 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105194.log: 88.7% -- replaced with /var/log/ceph/cluster2.log.gz 2023-12-02T07:22:27.051 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105701.log 2023-12-02T07:22:27.051 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101698.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103306.log.gz 2023-12-02T07:22:27.057 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101969.log 2023-12-02T07:22:27.057 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.105701.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101698.log.gz 2023-12-02T07:22:27.057 INFO:teuthology.orchestra.run.smithi057.stderr: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.105701.log.gz 2023-12-02T07:22:27.057 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111408.log 2023-12-02T07:22:27.058 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101969.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116297.log 2023-12-02T07:22:27.058 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101969.log.gz 2023-12-02T07:22:27.058 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.111408.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101932.log 2023-12-02T07:22:27.058 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116297.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.111408.log.gz 2023-12-02T07:22:27.058 INFO:teuthology.orchestra.run.smithi057.stderr: 10.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116297.log.gz 2023-12-02T07:22:27.058 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103871.log 2023-12-02T07:22:27.058 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101932.log: 79.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102590.log 2023-12-02T07:22:27.058 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.101932.log.gz 2023-12-02T07:22:27.059 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103871.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96062.log 2023-12-02T07:22:27.059 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102590.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103871.log.gz 2023-12-02T07:22:27.059 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102590.log.gz 2023-12-02T07:22:27.059 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110163.log 2023-12-02T07:22:27.059 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.96062.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100763.log 2023-12-02T07:22:27.060 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.110163.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.110163.log.gz 2023-12-02T07:22:27.060 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93943.log 2023-12-02T07:22:27.060 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100763.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100763.log.gz 2023-12-02T07:22:27.060 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115517.log 2023-12-02T07:22:27.061 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.93943.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.93943.log.gz 2023-12-02T07:22:27.061 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110545.log 2023-12-02T07:22:27.061 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115517.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115517.log.gz 2023-12-02T07:22:27.061 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98762.log 2023-12-02T07:22:27.063 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.110545.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110545.log.gz 2023-12-02T07:22:27.063 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109434.log 2023-12-02T07:22:27.064 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.98762.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.98762.log.gz 2023-12-02T07:22:27.064 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104449.log 2023-12-02T07:22:27.065 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.109434.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109434.log.gz 2023-12-02T07:22:27.065 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105016.log 2023-12-02T07:22:27.068 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104449.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104449.log.gz 2023-12-02T07:22:27.068 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.115982.log 2023-12-02T07:22:27.081 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105016.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95722.log 2023-12-02T07:22:27.082 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.115982.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.admin.115982.log.gz 2023-12-02T07:22:27.095 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113685.log 2023-12-02T07:22:27.097 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.95722.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.95722.log.gz 2023-12-02T07:22:27.097 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.91776.log 2023-12-02T07:22:27.098 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113685.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.113685.log.gz 2023-12-02T07:22:27.111 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112310.log 2023-12-02T07:22:27.125 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.3.91776.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115794.log 2023-12-02T07:22:27.132 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112310.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90381.log 2023-12-02T07:22:27.139 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.115794.log: 92.2% 94.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90507.log 2023-12-02T07:22:27.145 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.90381.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.admin.115794.log.gz 2023-12-02T07:22:27.149 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.112310.log.gz 2023-12-02T07:22:27.152 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.3.91776.log.gz 2023-12-02T07:22:27.152 INFO:teuthology.orchestra.run.smithi057.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90381.log.gz 2023-12-02T07:22:27.163 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110832.log 2023-12-02T07:22:27.163 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.90507.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90507.log.gz 2023-12-02T07:22:27.172 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111978.log 2023-12-02T07:22:27.173 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.110832.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.110832.log.gz 2023-12-02T07:22:27.185 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99630.log 2023-12-02T07:22:27.186 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.111978.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.111978.log.gz 2023-12-02T07:22:27.201 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104272.log 2023-12-02T07:22:27.207 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99630.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99630.log.gz 2023-12-02T07:22:27.218 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110629.log 2023-12-02T07:22:27.219 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104272.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.104272.log.gz 2023-12-02T07:22:27.227 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102533.log 2023-12-02T07:22:27.236 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.110629.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106918.log 2023-12-02T07:22:27.240 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102533.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.110629.log.gz 2023-12-02T07:22:27.245 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98405.log 2023-12-02T07:22:27.254 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.106918.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98133.log 2023-12-02T07:22:27.255 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.98405.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.98405.log.gz 2023-12-02T07:22:27.258 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.106918.log.gz 2023-12-02T07:22:27.272 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95814.log 2023-12-02T07:22:27.273 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.98133.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.98133.log.gz 2023-12-02T07:22:27.281 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95794.log 2023-12-02T07:22:27.290 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.95814.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112825.log 2023-12-02T07:22:27.294 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.95794.log: 89.4% 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.95814.log.gz 2023-12-02T07:22:27.299 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105044.log 2023-12-02T07:22:27.304 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112825.log: -- replaced with /var/log/ceph/cluster2-client.mirror.95794.log.gz 2023-12-02T07:22:27.308 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.102107.log 2023-12-02T07:22:27.312 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105044.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112825.log.gz 2023-12-02T07:22:27.319 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107664.log 2023-12-02T07:22:27.321 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.1.102107.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.105044.log.gz 2023-12-02T07:22:27.338 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110986.log 2023-12-02T07:22:27.339 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.107664.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.107664.log.gz 2023-12-02T07:22:27.347 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113105.log 2023-12-02T07:22:27.360 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.110986.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.110986.log.gz 2023-12-02T07:22:27.366 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106165.log 2023-12-02T07:22:27.367 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113105.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113105.log.gz 2023-12-02T07:22:27.375 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103247.log 2023-12-02T07:22:27.391 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.106165.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.105016.log.gz 2023-12-02T07:22:27.391 INFO:teuthology.orchestra.run.smithi057.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.106165.log.gz 2023-12-02T07:22:27.392 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101448.log.gz 2023-12-02T07:22:27.396 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99716.log 2023-12-02T07:22:27.396 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103247.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.103247.log.gz 2023-12-02T07:22:27.405 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.91673.log 2023-12-02T07:22:27.424 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99716.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.99716.log.gz 2023-12-02T07:22:27.427 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110255.log 2023-12-02T07:22:27.433 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.1.91673.log: 77.9% -- replaced with /var/log/ceph/cluster1-client.mirror.1.91673.log.gz 2023-12-02T07:22:27.437 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103466.log.gz 2023-12-02T07:22:27.439 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94731.log 2023-12-02T07:22:27.439 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.110255.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110255.log.gz 2023-12-02T07:22:27.440 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111049.log 2023-12-02T07:22:27.441 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.94731.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.94731.log.gz 2023-12-02T07:22:27.441 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90631.log 2023-12-02T07:22:27.442 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.111049.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.111049.log.gz 2023-12-02T07:22:27.442 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110745.log 2023-12-02T07:22:27.443 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.90631.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90631.log.gz 2023-12-02T07:22:27.444 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112385.log 2023-12-02T07:22:27.444 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.110745.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110745.log.gz 2023-12-02T07:22:27.445 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99942.log 2023-12-02T07:22:27.446 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112385.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112385.log.gz 2023-12-02T07:22:27.446 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103809.log 2023-12-02T07:22:27.447 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99942.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.99942.log.gz 2023-12-02T07:22:27.448 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107839.log 2023-12-02T07:22:27.450 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103809.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.96565.log.gz 2023-12-02T07:22:27.451 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98263.log 2023-12-02T07:22:27.454 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.107839.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.107839.log.gz 2023-12-02T07:22:27.454 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90756.log 2023-12-02T07:22:27.455 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.98263.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.98263.log.gz 2023-12-02T07:22:27.455 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109809.log 2023-12-02T07:22:27.456 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.90756.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90756.log.gz 2023-12-02T07:22:27.456 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102553.log 2023-12-02T07:22:27.458 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.109809.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.109809.log.gz 2023-12-02T07:22:27.458 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91557.log 2023-12-02T07:22:27.459 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102553.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102553.log.gz 2023-12-02T07:22:27.459 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98037.log 2023-12-02T07:22:27.459 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.91557.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.91557.log.gz 2023-12-02T07:22:27.461 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99458.log 2023-12-02T07:22:27.461 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.98037.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.98037.log.gz 2023-12-02T07:22:27.462 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109458.log 2023-12-02T07:22:27.467 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99458.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99458.log.gz 2023-12-02T07:22:27.472 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99842.log 2023-12-02T07:22:27.472 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.109458.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109458.log.gz 2023-12-02T07:22:27.472 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102570.log 2023-12-02T07:22:27.472 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99842.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.99842.log.gz 2023-12-02T07:22:27.472 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110183.log 2023-12-02T07:22:27.472 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102570.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102570.log.gz 2023-12-02T07:22:27.475 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113025.log 2023-12-02T07:22:27.475 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.110183.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.110183.log.gz 2023-12-02T07:22:27.476 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91776.log 2023-12-02T07:22:27.476 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113025.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.113025.log.gz 2023-12-02T07:22:27.476 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93472.log 2023-12-02T07:22:27.489 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.91776.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107859.log 2023-12-02T07:22:27.508 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.93472.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111793.log 2023-12-02T07:22:27.510 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.107859.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.107859.log.gz 2023-12-02T07:22:27.516 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114745.log 2023-12-02T07:22:27.516 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.111793.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.111793.log.gz 2023-12-02T07:22:27.527 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116371.log 2023-12-02T07:22:27.528 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114745.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114745.log.gz 2023-12-02T07:22:27.532 INFO:teuthology.orchestra.run.smithi057.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.91776.log.gz 2023-12-02T07:22:27.548 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101426.log 2023-12-02T07:22:27.548 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116371.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116371.log.gz 2023-12-02T07:22:27.550 INFO:teuthology.orchestra.run.smithi057.stderr: 94.4% -- replaced with /var/log/ceph/cluster2-client.mirror.93472.log.gz 2023-12-02T07:22:27.551 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108964.log 2023-12-02T07:22:27.551 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101426.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101426.log.gz 2023-12-02T07:22:27.558 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116328.log 2023-12-02T07:22:27.558 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.108964.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108964.log.gz 2023-12-02T07:22:27.558 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99982.log 2023-12-02T07:22:27.559 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116328.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116328.log.gz 2023-12-02T07:22:27.559 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107435.log 2023-12-02T07:22:27.559 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99982.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.99982.log.gz 2023-12-02T07:22:27.559 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101718.log 2023-12-02T07:22:27.559 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.107435.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.107435.log.gz 2023-12-02T07:22:27.560 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109328.log 2023-12-02T07:22:27.560 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101718.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101718.log.gz 2023-12-02T07:22:27.560 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105389.log 2023-12-02T07:22:27.560 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.96062.log.gz/var/log/ceph/cluster2-client.mirror.109328.log: 2023-12-02T07:22:27.560 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.109328.log.gz 2023-12-02T07:22:27.560 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.114965.log 2023-12-02T07:22:27.560 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110026.log 2023-12-02T07:22:27.627 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105389.log: /var/log/ceph/cluster1-client.mirror.114965.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114965.log.gz 2023-12-02T07:22:27.628 INFO:teuthology.orchestra.run.smithi057.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.105389.log.gz 2023-12-02T07:22:27.628 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100935.log 2023-12-02T07:22:27.628 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.110026.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107577.log 2023-12-02T07:22:27.628 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.110026.log.gz 2023-12-02T07:22:27.628 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100935.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100935.log.gz 2023-12-02T07:22:27.628 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101039.log 2023-12-02T07:22:27.628 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.107577.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94607.log 2023-12-02T07:22:27.629 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.107577.log.gz 2023-12-02T07:22:27.629 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101039.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101039.log.gz 2023-12-02T07:22:27.629 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115923.log 2023-12-02T07:22:27.629 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111228.log 2023-12-02T07:22:27.629 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.94607.log: /var/log/ceph/cluster1-client.mirror.115923.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115923.log.gz 2023-12-02T07:22:27.629 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90843.log 2023-12-02T07:22:27.629 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.111228.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.111228.log.gz 2023-12-02T07:22:27.629 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113205.log 2023-12-02T07:22:27.630 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.90843.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90843.log.gz 2023-12-02T07:22:27.630 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113445.log 2023-12-02T07:22:27.630 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113205.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113205.log.gz 2023-12-02T07:22:27.630 INFO:teuthology.orchestra.run.smithi057.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.94607.log.gz 2023-12-02T07:22:27.630 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103671.log 2023-12-02T07:22:27.630 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108077.log 2023-12-02T07:22:27.630 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113445.log: /var/log/ceph/cluster2-client.mirror.103671.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113445.log.gz 2023-12-02T07:22:27.630 INFO:teuthology.orchestra.run.smithi057.stderr: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103671.log.gz 2023-12-02T07:22:27.630 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107372.log 2023-12-02T07:22:27.631 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90549.log 2023-12-02T07:22:27.631 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.108077.log: /var/log/ceph/cluster1-client.mirror.107372.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.107372.log.gz 2023-12-02T07:22:27.631 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100296.log 2023-12-02T07:22:27.631 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.90549.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90549.log.gz 2023-12-02T07:22:27.631 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92222.log 2023-12-02T07:22:27.631 INFO:teuthology.orchestra.run.smithi057.stderr: 92.4%/var/log/ceph/cluster2-client.mirror.100296.log: -- replaced with /var/log/ceph/cluster1-client.mirror.108077.log.gz 2023-12-02T07:22:27.631 INFO:teuthology.orchestra.run.smithi057.stderr: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100296.log.gz 2023-12-02T07:22:27.631 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98824.log 2023-12-02T07:22:27.631 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.admin.92222.log: -- /var/log/ceph/cluster2-client.mirror.1.91847.log 2023-12-02T07:22:27.632 INFO:teuthology.orchestra.run.smithi057.stderr: 66.9% -- replaced with /var/log/ceph/cluster1-client.admin.92222.log.gz 2023-12-02T07:22:27.632 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.98824.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103708.log 2023-12-02T07:22:27.632 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.1.91847.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.98824.log.gz 2023-12-02T07:22:27.632 INFO:teuthology.orchestra.run.smithi057.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.1.91847.log.gz 2023-12-02T07:22:27.632 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110404.log 2023-12-02T07:22:27.632 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103708.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92384.log 2023-12-02T07:22:27.632 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103708.log.gz 2023-12-02T07:22:27.632 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.110404.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114625.log 2023-12-02T07:22:27.632 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.92384.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.110404.log.gz 2023-12-02T07:22:27.633 INFO:teuthology.orchestra.run.smithi057.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.92384.log.gz 2023-12-02T07:22:27.633 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100972.log 2023-12-02T07:22:27.633 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.114625.log: -- /var/log/ceph/cluster1-client.mirror.98157.log 2023-12-02T07:22:27.633 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.114625.log.gz 2023-12-02T07:22:27.633 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100972.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113605.log 2023-12-02T07:22:27.633 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.98157.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.100972.log.gz 2023-12-02T07:22:27.633 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99134.log 2023-12-02T07:22:27.633 INFO:teuthology.orchestra.run.smithi057.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98157.log.gz 2023-12-02T07:22:27.633 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113605.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99243.log 2023-12-02T07:22:27.634 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.99134.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.113605.log.gz 2023-12-02T07:22:27.634 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99882.log 2023-12-02T07:22:27.634 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99243.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.99134.log.gz 2023-12-02T07:22:27.634 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107529.log 2023-12-02T07:22:27.634 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99882.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99243.log.gz 2023-12-02T07:22:27.634 INFO:teuthology.orchestra.run.smithi057.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.99882.log.gz 2023-12-02T07:22:27.634 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104957.log 2023-12-02T07:22:27.634 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115572.log/var/log/ceph/cluster1-client.mirror.107529.log: 2023-12-02T07:22:27.635 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107529.log.gz 2023-12-02T07:22:27.635 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104957.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104957.log.gz 2023-12-02T07:22:27.635 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111111.log 2023-12-02T07:22:27.635 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108671.log 2023-12-02T07:22:27.635 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.111111.log: /var/log/ceph/cluster1-client.admin.115572.log: 70.4% -- replaced with /var/log/ceph/cluster1-client.admin.115572.log.gz 2023-12-02T07:22:27.635 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.111111.log.gz 2023-12-02T07:22:27.635 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111998.log 2023-12-02T07:22:27.635 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106257.log 2023-12-02T07:22:27.635 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.111998.log: /var/log/ceph/cluster2-client.mirror.108671.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.108671.log.gz 2023-12-02T07:22:27.636 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.111998.log.gz 2023-12-02T07:22:27.636 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92734.log 2023-12-02T07:22:27.636 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114125.log 2023-12-02T07:22:27.636 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.106257.log: /var/log/ceph/cluster2-client.mirror.92734.log: 4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.92734.log.gz 2023-12-02T07:22:27.636 INFO:teuthology.orchestra.run.smithi057.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.106257.log.gz 2023-12-02T07:22:27.636 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.116496.log 2023-12-02T07:22:27.636 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112805.log 2023-12-02T07:22:27.636 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.116496.log: /var/log/ceph/cluster1-client.mirror.114125.log: 9.2% -- replaced with /var/log/ceph/cluster1-client.admin.116496.log.gz 2023-12-02T07:22:27.637 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.114125.log.gz 2023-12-02T07:22:27.637 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103750.log 2023-12-02T07:22:27.637 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112805.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114785.log 2023-12-02T07:22:27.637 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.112805.log.gz 2023-12-02T07:22:27.637 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103750.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103750.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114225.log 2023-12-02T07:22:27.637 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-12-02T07:22:27.637 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114785.log: gzip 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114785.log.gz 2023-12-02T07:22:27.637 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113625.log 2023-12-02T07:22:27.638 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113945.log 2023-12-02T07:22:27.638 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114225.log: /var/log/ceph/cluster1-client.mirror.113625.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.114225.log.gz 2023-12-02T07:22:27.638 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113625.log.gz 2023-12-02T07:22:27.638 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113065.log 2023-12-02T07:22:27.638 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106894.log 2023-12-02T07:22:27.638 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.113945.log.gz 2023-12-02T07:22:27.638 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113065.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99415.log 2023-12-02T07:22:27.638 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.113065.log.gz 2023-12-02T07:22:27.639 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.106894.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113965.log 2023-12-02T07:22:27.639 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99415.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.106894.log.gz 2023-12-02T07:22:27.639 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99607.log 2023-12-02T07:22:27.639 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113965.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113965.log.gz 2023-12-02T07:22:27.639 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110649.log 2023-12-02T07:22:27.639 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.99607.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99415.log.gz 2023-12-02T07:22:27.639 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112665.log 2023-12-02T07:22:27.639 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.110649.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110649.log.gz 2023-12-02T07:22:27.639 INFO:teuthology.orchestra.run.smithi057.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99607.log.gz 2023-12-02T07:22:27.640 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105330.log 2023-12-02T07:22:27.640 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108892.log 2023-12-02T07:22:27.640 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105330.log: /var/log/ceph/cluster1-client.mirror.112665.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112665.log.gz 2023-12-02T07:22:27.640 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.105330.log.gz 2023-12-02T07:22:27.640 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92164.log 2023-12-02T07:22:27.640 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109587.log 2023-12-02T07:22:27.640 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.108892.log: /var/log/ceph/cluster1-client.admin.92164.log: 0.0% 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.108892.log.gz 2023-12-02T07:22:27.640 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.92164.log.gz 2023-12-02T07:22:27.640 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96388.log 2023-12-02T07:22:27.641 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110050.log 2023-12-02T07:22:27.641 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.109587.log: /var/log/ceph/cluster1-client.mirror.96388.log: 88.8% 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.109587.log.gz 2023-12-02T07:22:27.641 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.96388.log.gz 2023-12-02T07:22:27.641 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104315.log 2023-12-02T07:22:27.641 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95045.log 2023-12-02T07:22:27.641 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.110050.log: /var/log/ceph/cluster2-client.mirror.104315.log: 79.5% 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.110050.log.gz 2023-12-02T07:22:27.641 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.104315.log.gz 2023-12-02T07:22:27.641 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95171.log 2023-12-02T07:22:27.641 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102320.log 2023-12-02T07:22:27.642 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.95045.log: /var/log/ceph/cluster2-client.mirror.95171.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95171.log.gz 2023-12-02T07:22:27.642 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116509.log 2023-12-02T07:22:27.642 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102320.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102320.log.gz 2023-12-02T07:22:27.642 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113585.log 2023-12-02T07:22:27.642 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.116509.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.116509.log.gz 2023-12-02T07:22:27.642 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90883.log 2023-12-02T07:22:27.642 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113585.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113585.log.gz 2023-12-02T07:22:27.642 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100142.log 2023-12-02T07:22:27.643 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.90883.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90883.log.gz 2023-12-02T07:22:27.643 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104756.log 2023-12-02T07:22:27.643 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100142.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100142.log.gz 2023-12-02T07:22:27.643 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103992.log 2023-12-02T07:22:27.644 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104756.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105616.log 2023-12-02T07:22:27.644 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103992.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103992.log.gz 2023-12-02T07:22:27.650 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92768.log 2023-12-02T07:22:27.659 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105616.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114885.log 2023-12-02T07:22:27.668 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.92768.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115871.log 2023-12-02T07:22:27.668 INFO:teuthology.orchestra.run.smithi057.stderr: 57.7% -- replaced with /var/log/ceph/cluster2-client.mirror.92768.log.gz/var/log/ceph/cluster1-client.mirror.114885.log: 2023-12-02T07:22:27.681 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116422.log 2023-12-02T07:22:27.681 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114885.log.gz 2023-12-02T07:22:27.682 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115871.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115871.log.gz 2023-12-02T07:22:27.697 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115479.log 2023-12-02T07:22:27.697 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116422.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116422.log.gz 2023-12-02T07:22:27.706 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101161.log 2023-12-02T07:22:27.706 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115479.log: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.115479.log.gz 2023-12-02T07:22:27.715 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105875.log 2023-12-02T07:22:27.731 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101161.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102533.log.gz 2023-12-02T07:22:27.733 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101885.log 2023-12-02T07:22:27.733 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.105875.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.105875.log.gz 2023-12-02T07:22:27.745 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110914.log 2023-12-02T07:22:27.745 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101885.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101885.log.gz 2023-12-02T07:22:27.757 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103205.log 2023-12-02T07:22:27.758 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.110914.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.110914.log.gz 2023-12-02T07:22:27.771 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98017.log 2023-12-02T07:22:27.771 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103205.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103205.log.gz 2023-12-02T07:22:27.785 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106349.log 2023-12-02T07:22:27.786 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.98017.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98017.log.gz 2023-12-02T07:22:27.794 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112945.log 2023-12-02T07:22:27.808 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.106349.log: gzip -5 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.106349.log.gz 2023-12-02T07:22:27.808 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.106870.log 2023-12-02T07:22:27.808 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112945.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.112945.log.gz 2023-12-02T07:22:27.816 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.104608.log 2023-12-02T07:22:27.829 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.106870.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106870.log.gz 2023-12-02T07:22:27.830 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103503.log 2023-12-02T07:22:27.839 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.3.104608.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98920.log 2023-12-02T07:22:27.851 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.103503.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98632.log 2023-12-02T07:22:27.852 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.98920.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.98920.log.gz 2023-12-02T07:22:27.854 INFO:teuthology.orchestra.run.smithi057.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.mirror.103503.log.gz 2023-12-02T07:22:27.861 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97566.log 2023-12-02T07:22:27.862 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.98632.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98632.log.gz 2023-12-02T07:22:27.877 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90757.log 2023-12-02T07:22:27.889 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.97566.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112157.log 2023-12-02T07:22:27.889 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.90757.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90757.log.gz 2023-12-02T07:22:27.893 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.97566.log.gz 2023-12-02T07:22:27.897 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98285.log 2023-12-02T07:22:27.905 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.112157.log: 90.1% 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.105194.log.gz 2023-12-02T07:22:27.906 INFO:teuthology.orchestra.run.smithi057.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.112157.log.gz -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2023-12-02T07:22:27.906 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-12-02T07:22:27.914 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104843.log 2023-12-02T07:22:27.924 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.98285.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94564.log 2023-12-02T07:22:27.934 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104843.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94345.log 2023-12-02T07:22:27.948 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.94564.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115590.log 2023-12-02T07:22:27.948 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.94345.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.94345.log.gz 2023-12-02T07:22:27.953 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.94564.log.gz 2023-12-02T07:22:27.960 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91691.log 2023-12-02T07:22:27.961 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115590.log: 68.1% -- replaced with /var/log/ceph/cluster1-client.mirror.115590.log.gz 2023-12-02T07:22:27.969 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105273.log 2023-12-02T07:22:27.988 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.91691.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110500.log 2023-12-02T07:22:28.005 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105273.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111207.log 2023-12-02T07:22:28.005 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.110500.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.110500.log.gz 2023-12-02T07:22:28.017 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107168.log 2023-12-02T07:22:28.018 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.111207.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111207.log.gz 2023-12-02T07:22:28.031 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98942.log 2023-12-02T07:22:28.033 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.107168.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.107168.log.gz 2023-12-02T07:22:28.050 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111270.log 2023-12-02T07:22:28.056 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.98942.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.98942.log.gz 2023-12-02T07:22:28.063 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99372.log 2023-12-02T07:22:28.064 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.111270.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.111270.log.gz 2023-12-02T07:22:28.077 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108651.log 2023-12-02T07:22:28.079 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99372.log: gzip: /var/log/ceph/cluster1-client.mirror.1.102107.log: file size changed while zipping 2023-12-02T07:22:28.079 INFO:teuthology.orchestra.run.smithi057.stderr: 95.6% -- replaced with /var/log/ceph/cluster1-client.mirror.1.102107.log.gz 2023-12-02T07:22:28.080 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97626.log 2023-12-02T07:22:28.082 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.108651.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.108651.log.gz 2023-12-02T07:22:28.082 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108418.log 2023-12-02T07:22:28.083 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.97626.log: 74.4% -- replaced with /var/log/ceph/cluster1-client.mirror.97626.log.gz 2023-12-02T07:22:28.083 INFO:teuthology.orchestra.run.smithi057.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.99372.log.gz 2023-12-02T07:22:28.088 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103809.log.gz 2023-12-02T07:22:28.094 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104756.log.gz 2023-12-02T07:22:28.103 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108099.log 2023-12-02T07:22:28.104 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.108418.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.108418.log.gz 2023-12-02T07:22:28.113 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102048.log 2023-12-02T07:22:28.114 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.108099.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.108099.log.gz 2023-12-02T07:22:28.116 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.105616.log.gz 2023-12-02T07:22:28.120 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110476.log 2023-12-02T07:22:28.133 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102048.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102048.log.gz 2023-12-02T07:22:28.135 INFO:teuthology.orchestra.run.smithi057.stderr: 94.4% -- replaced with /var/log/ceph/cluster2-client.mirror.91691.log.gz 2023-12-02T07:22:28.136 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106822.log 2023-12-02T07:22:28.137 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.110476.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110476.log.gz 2023-12-02T07:22:28.137 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107348.log 2023-12-02T07:22:28.138 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.106822.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.106822.log.gz 2023-12-02T07:22:28.138 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102427.log 2023-12-02T07:22:28.139 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.107348.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107348.log.gz 2023-12-02T07:22:28.139 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107642.log 2023-12-02T07:22:28.140 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102427.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.102427.log.gz 2023-12-02T07:22:28.140 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96910.log 2023-12-02T07:22:28.142 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.107642.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.107642.log.gz 2023-12-02T07:22:28.143 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113925.log 2023-12-02T07:22:28.143 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.96910.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.96910.log.gz 2023-12-02T07:22:28.144 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108036.log 2023-12-02T07:22:28.146 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113925.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.113925.log.gz 2023-12-02T07:22:28.146 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108265.log 2023-12-02T07:22:28.147 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.108036.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.108036.log.gz 2023-12-02T07:22:28.147 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111159.log 2023-12-02T07:22:28.149 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.108265.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.108265.log.gz 2023-12-02T07:22:28.149 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116583.log 2023-12-02T07:22:28.151 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.111159.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111159.log.gz 2023-12-02T07:22:28.151 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96018.log 2023-12-02T07:22:28.151 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.116583.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.116583.log.gz 2023-12-02T07:22:28.151 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106479.log 2023-12-02T07:22:28.157 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.96018.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.96018.log.gz 2023-12-02T07:22:28.158 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95125.log 2023-12-02T07:22:28.159 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.106479.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.106479.log.gz 2023-12-02T07:22:28.159 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.105426.log 2023-12-02T07:22:28.161 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.95125.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95125.log.gz 2023-12-02T07:22:28.161 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104051.log 2023-12-02T07:22:28.174 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.2.105426.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108788.log 2023-12-02T07:22:28.181 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104051.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104994.log 2023-12-02T07:22:28.188 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.108788.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106096.log 2023-12-02T07:22:28.190 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104994.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.108788.log.gz 2023-12-02T07:22:28.191 INFO:teuthology.orchestra.run.smithi057.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.2.105426.log.gz 2023-12-02T07:22:28.201 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.104994.log.gz 2023-12-02T07:22:28.204 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95910.log 2023-12-02T07:22:28.204 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.106096.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.106096.log.gz 2023-12-02T07:22:28.218 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102871.log 2023-12-02T07:22:28.218 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.95910.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.95910.log.gz 2023-12-02T07:22:28.232 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91305.log 2023-12-02T07:22:28.242 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102871.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114705.log 2023-12-02T07:22:28.242 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.91305.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.91305.log.gz 2023-12-02T07:22:28.244 INFO:teuthology.orchestra.run.smithi057.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.102871.log.gz 2023-12-02T07:22:28.258 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107051.log 2023-12-02T07:22:28.259 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114705.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.114705.log.gz 2023-12-02T07:22:28.284 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112605.log 2023-12-02T07:22:28.284 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.107051.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.107051.log.gz 2023-12-02T07:22:28.293 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115940.log 2023-12-02T07:22:28.293 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112605.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.112605.log.gz 2023-12-02T07:22:28.308 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115777.log 2023-12-02T07:22:28.308 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115940.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101161.log.gz 2023-12-02T07:22:28.308 INFO:teuthology.orchestra.run.smithi057.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115940.log.gz 2023-12-02T07:22:28.328 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103913.log 2023-12-02T07:22:28.328 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115777.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115777.log.gz 2023-12-02T07:22:28.333 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105944.log 2023-12-02T07:22:28.333 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103913.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103913.log.gz 2023-12-02T07:22:28.352 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99286.log 2023-12-02T07:22:28.354 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105944.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.105944.log.gz 2023-12-02T07:22:28.370 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2023-12-02T07:22:28.375 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99286.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99286.log.gz 2023-12-02T07:22:28.380 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116142.log 2023-12-02T07:22:28.389 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114405.log 2023-12-02T07:22:28.390 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116142.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116142.log.gz 2023-12-02T07:22:28.400 INFO:teuthology.orchestra.run.smithi057.stderr: 92.7% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2023-12-02T07:22:28.405 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93967.log 2023-12-02T07:22:28.405 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114405.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.114405.log.gz 2023-12-02T07:22:28.419 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114065.log 2023-12-02T07:22:28.420 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.93967.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.93967.log.gz 2023-12-02T07:22:28.440 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109125.log 2023-12-02T07:22:28.445 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114065.log: 87.4% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.95045.log.gz 2023-12-02T07:22:28.446 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95345.log 2023-12-02T07:22:28.447 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.109125.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.109125.log.gz 2023-12-02T07:22:28.448 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109743.log 2023-12-02T07:22:28.449 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.95345.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95345.log.gz 2023-12-02T07:22:28.450 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114205.log 2023-12-02T07:22:28.451 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.109743.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109743.log.gz 2023-12-02T07:22:28.452 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115145.log 2023-12-02T07:22:28.452 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114205.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.114205.log.gz 2023-12-02T07:22:28.453 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106537.log 2023-12-02T07:22:28.454 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115145.log: -- replaced with /var/log/ceph/cluster1-client.mirror.114065.log.gz 2023-12-02T07:22:28.455 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.115145.log.gz 2023-12-02T07:22:28.455 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113865.log 2023-12-02T07:22:28.458 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.106537.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.106537.log.gz 2023-12-02T07:22:28.458 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114765.log 2023-12-02T07:22:28.459 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113865.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.113865.log.gz 2023-12-02T07:22:28.459 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111027.log 2023-12-02T07:22:28.460 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114765.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.114765.log.gz 2023-12-02T07:22:28.461 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104734.log 2023-12-02T07:22:28.463 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.111027.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.111027.log.gz 2023-12-02T07:22:28.464 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99650.log 2023-12-02T07:22:28.465 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104734.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.104734.log.gz 2023-12-02T07:22:28.466 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105103.log 2023-12-02T07:22:28.472 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.99650.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.99650.log.gz 2023-12-02T07:22:28.473 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112368.log 2023-12-02T07:22:28.487 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105103.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113305.log 2023-12-02T07:22:28.487 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112368.log: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.112368.log.gz 2023-12-02T07:22:28.499 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.105273.log.gz 2023-12-02T07:22:28.500 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109607.log 2023-12-02T07:22:28.501 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113305.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.113305.log.gz 2023-12-02T07:22:28.502 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113985.log 2023-12-02T07:22:28.503 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.109607.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.109607.log.gz 2023-12-02T07:22:28.503 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90629.log 2023-12-02T07:22:28.506 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113985.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.113985.log.gz 2023-12-02T07:22:28.506 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115649.log 2023-12-02T07:22:28.506 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.90629.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90629.log.gz 2023-12-02T07:22:28.506 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105310.log 2023-12-02T07:22:28.506 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115649.log: 10.2% -- replaced with /var/log/ceph/cluster1-client.mirror.115649.log.gz 2023-12-02T07:22:28.506 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103126.log 2023-12-02T07:22:28.510 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105310.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.105310.log.gz 2023-12-02T07:22:28.510 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- 95.9% -- replaced with /var/log/ceph/cluster1-client.mirror.3.104608.log.gz 2023-12-02T07:22:28.510 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/cluster2-client.mirror.116129.log 2023-12-02T07:22:28.510 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103126.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116075.log 2023-12-02T07:22:28.510 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103126.log.gz 2023-12-02T07:22:28.511 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116129.log: 9.2%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109053.log 2023-12-02T07:22:28.511 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.116129.log.gz/var/log/ceph/cluster2-client.mirror.116075.log: 2023-12-02T07:22:28.511 INFO:teuthology.orchestra.run.smithi057.stderr: 7.1% -- replaced with /var/log/ceph/cluster2-client.mirror.116075.log.gz 2023-12-02T07:22:28.511 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111432.log 2023-12-02T07:22:28.511 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97501.log 2023-12-02T07:22:28.512 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.109053.log: /var/log/ceph/cluster1-client.mirror.111432.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.111432.log.gz 2023-12-02T07:22:28.512 INFO:teuthology.orchestra.run.smithi057.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109053.log.gz 2023-12-02T07:22:28.512 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103106.log 2023-12-02T07:22:28.513 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102982.log 2023-12-02T07:22:28.514 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.97501.log: /var/log/ceph/cluster2-client.mirror.103106.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.97501.log.gz 2023-12-02T07:22:28.514 INFO:teuthology.orchestra.run.smithi057.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112505.log 2023-12-02T07:22:28.515 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.103106.log.gz 2023-12-02T07:22:28.515 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102982.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.102649.log 2023-12-02T07:22:28.515 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112505.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112505.log.gz 2023-12-02T07:22:28.515 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101139.log 2023-12-02T07:22:28.528 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.3.102649.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108330.log 2023-12-02T07:22:28.529 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101139.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101139.log.gz 2023-12-02T07:22:28.541 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112785.log 2023-12-02T07:22:28.543 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.108330.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108330.log.gz 2023-12-02T07:22:28.554 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98178.log 2023-12-02T07:22:28.555 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112785.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112785.log.gz 2023-12-02T07:22:28.564 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104843.log.gz 2023-12-02T07:22:28.565 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103972.log 2023-12-02T07:22:28.571 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.98178.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98178.log.gz 2023-12-02T07:22:28.572 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115345.log 2023-12-02T07:22:28.581 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103972.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95692.log 2023-12-02T07:22:28.594 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115345.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105293.log 2023-12-02T07:22:28.594 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.95692.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.115345.log.gz 2023-12-02T07:22:28.595 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.95692.log.gz 2023-12-02T07:22:28.609 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104252.log 2023-12-02T07:22:28.609 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105293.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.105293.log.gz 2023-12-02T07:22:28.618 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99071.log 2023-12-02T07:22:28.620 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104252.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104252.log.gz 2023-12-02T07:22:28.624 INFO:teuthology.orchestra.run.smithi057.stderr: 95.0% -- replaced with /var/log/ceph/cluster2-client.mirror.3.102649.log.gz 2023-12-02T07:22:28.625 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108169.log 2023-12-02T07:22:28.631 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99071.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99071.log.gz 2023-12-02T07:22:28.632 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109075.log 2023-12-02T07:22:28.634 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.108169.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.108169.log.gz 2023-12-02T07:22:28.634 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92239.log 2023-12-02T07:22:28.634 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.109075.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.109075.log.gz 2023-12-02T07:22:28.635 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109954.log 2023-12-02T07:22:28.636 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.92239.log: 54.4% -- replaced with /var/log/ceph/cluster1-client.mirror.92239.log.gz 2023-12-02T07:22:28.637 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109934.log 2023-12-02T07:22:28.637 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.109954.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.109954.log.gz 2023-12-02T07:22:28.638 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99802.log 2023-12-02T07:22:28.639 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.109934.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.109934.log.gz 2023-12-02T07:22:28.640 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108352.log 2023-12-02T07:22:28.641 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99802.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99802.log.gz 2023-12-02T07:22:28.642 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94651.log 2023-12-02T07:22:28.642 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.108352.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.108352.log.gz 2023-12-02T07:22:28.643 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106277.log 2023-12-02T07:22:28.661 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.94651.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109012.log 2023-12-02T07:22:28.661 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.106277.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.106277.log.gz 2023-12-02T07:22:28.663 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111749.log 2023-12-02T07:22:28.663 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.109012.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.109012.log.gz 2023-12-02T07:22:28.664 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104697.log 2023-12-02T07:22:28.664 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.111749.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.111749.log.gz 2023-12-02T07:22:28.674 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100162.log 2023-12-02T07:22:28.674 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104697.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.104697.log.gz 2023-12-02T07:22:28.674 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103950.log 2023-12-02T07:22:28.675 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100162.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100162.log.gz 2023-12-02T07:22:28.675 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107300.log 2023-12-02T07:22:28.675 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103950.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103950.log.gz 2023-12-02T07:22:28.675 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107147.log 2023-12-02T07:22:28.675 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.107300.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107300.log.gz 2023-12-02T07:22:28.675 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115760.log 2023-12-02T07:22:28.675 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.107147.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.107147.log.gz 2023-12-02T07:22:28.675 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.112262.log 2023-12-02T07:22:28.675 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115760.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115760.log.gz 2023-12-02T07:22:28.676 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98584.log 2023-12-02T07:22:28.676 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.112262.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.112262.log.gz 2023-12-02T07:22:28.676 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113845.log 2023-12-02T07:22:28.676 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.98584.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.98584.log.gz 2023-12-02T07:22:28.676 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101819.log 2023-12-02T07:22:28.676 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113845.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113845.log.gz 2023-12-02T07:22:28.677 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97404.log 2023-12-02T07:22:28.689 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101819.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111817.log 2023-12-02T07:22:28.689 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.97404.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.97404.log.gz 2023-12-02T07:22:28.700 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99048.log 2023-12-02T07:22:28.700 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7%/var/log/ceph/cluster1-client.mirror.111817.log: -- replaced with /var/log/ceph/cluster2-client.mirror.98285.log.gz 2023-12-02T07:22:28.700 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.111817.log.gz 2023-12-02T07:22:28.700 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102474.log 2023-12-02T07:22:28.705 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.99048.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99048.log.gz 2023-12-02T07:22:28.707 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92188.log 2023-12-02T07:22:28.707 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102474.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102474.log.gz 2023-12-02T07:22:28.708 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110113.log 2023-12-02T07:22:28.708 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.92188.log: 80.1% -- replaced with /var/log/ceph/cluster1-client.admin.92188.log.gz 2023-12-02T07:22:28.708 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109386.log 2023-12-02T07:22:28.712 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.110113.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110113.log.gz 2023-12-02T07:22:28.712 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107553.log 2023-12-02T07:22:28.712 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.109386.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.109386.log.gz 2023-12-02T07:22:28.712 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113185.log 2023-12-02T07:22:28.712 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.107553.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.107553.log.gz 2023-12-02T07:22:28.712 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94277.log 2023-12-02T07:22:28.713 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113185.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.113185.log.gz 2023-12-02T07:22:28.713 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92858.log 2023-12-02T07:22:28.715 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.94277.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.94277.log.gz 2023-12-02T07:22:28.715 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109833.log 2023-12-02T07:22:28.715 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.92858.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.92858.log.gz 2023-12-02T07:22:28.715 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90882.log 2023-12-02T07:22:28.715 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.109833.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.109833.log.gz 2023-12-02T07:22:28.717 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99736.log 2023-12-02T07:22:28.717 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.90882.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90882.log.gz 2023-12-02T07:22:28.717 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109874.log 2023-12-02T07:22:28.722 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.99736.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99736.log.gz 2023-12-02T07:22:28.723 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95998.log 2023-12-02T07:22:28.724 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.109874.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.109874.log.gz 2023-12-02T07:22:28.725 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108057.log 2023-12-02T07:22:28.730 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.95998.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95998.log.gz 2023-12-02T07:22:28.731 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104548.log 2023-12-02T07:22:28.734 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.108057.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.108057.log.gz 2023-12-02T07:22:28.734 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.95538.log 2023-12-02T07:22:28.734 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104548.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.104548.log.gz 2023-12-02T07:22:28.735 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113425.log 2023-12-02T07:22:28.748 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.3.95538.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102817.log 2023-12-02T07:22:28.749 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113425.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.113425.log.gz 2023-12-02T07:22:28.752 INFO:teuthology.orchestra.run.smithi057.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.3.95538.log.gz 2023-12-02T07:22:28.753 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116401.log 2023-12-02T07:22:28.753 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102817.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102817.log.gz 2023-12-02T07:22:28.755 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102196.log 2023-12-02T07:22:28.755 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116401.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116401.log.gz 2023-12-02T07:22:28.756 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94126.log 2023-12-02T07:22:28.757 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102196.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102196.log.gz 2023-12-02T07:22:28.757 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80677.log 2023-12-02T07:22:28.759 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.94126.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.94126.log.gz 2023-12-02T07:22:28.759 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113805.log 2023-12-02T07:22:28.759 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph.tmp-client.admin.80677.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.80677.log.gz 2023-12-02T07:22:28.760 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94321.log 2023-12-02T07:22:28.761 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113805.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113805.log.gz 2023-12-02T07:22:28.761 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112094.log 2023-12-02T07:22:28.761 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.94321.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.94321.log.gz 2023-12-02T07:22:28.763 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.91691.log 2023-12-02T07:22:28.763 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112094.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.112094.log.gz 2023-12-02T07:22:28.764 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107324.log 2023-12-02T07:22:28.765 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.2.91691.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101907.log 2023-12-02T07:22:28.766 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.107324.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.107324.log.gz 2023-12-02T07:22:28.766 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99200.log 2023-12-02T07:22:28.775 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101907.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104051.log.gz 2023-12-02T07:22:28.776 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113785.log 2023-12-02T07:22:28.781 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99200.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99200.log.gz 2023-12-02T07:22:28.782 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107940.log 2023-12-02T07:22:28.784 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113785.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113785.log.gz 2023-12-02T07:22:28.784 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116286.log 2023-12-02T07:22:28.785 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.107940.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.107940.log.gz 2023-12-02T07:22:28.785 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91389.log 2023-12-02T07:22:28.787 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116286.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.mirror.116286.log.gz 2023-12-02T07:22:28.787 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116053.log 2023-12-02T07:22:28.787 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.91389.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.91389.log.gz 2023-12-02T07:22:28.789 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100693.log 2023-12-02T07:22:28.789 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116053.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116053.log.gz 2023-12-02T07:22:28.789 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95069.log 2023-12-02T07:22:28.790 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100693.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100693.log.gz 2023-12-02T07:22:28.790 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95438.log 2023-12-02T07:22:28.791 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.95069.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.95069.log.gz 2023-12-02T07:22:28.792 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114185.log 2023-12-02T07:22:28.798 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.95438.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95438.log.gz 2023-12-02T07:22:28.798 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98109.log 2023-12-02T07:22:28.800 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114185.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.114185.log.gz 2023-12-02T07:22:28.800 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2023-12-02T07:22:28.801 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.98109.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.98109.log.gz 2023-12-02T07:22:28.801 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103284.log 2023-12-02T07:22:28.802 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115718.log 2023-12-02T07:22:28.803 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103284.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103284.log.gz 2023-12-02T07:22:28.816 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114725.log 2023-12-02T07:22:28.817 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115718.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115718.log.gz 2023-12-02T07:22:28.823 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113165.log 2023-12-02T07:22:28.824 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114725.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.114725.log.gz 2023-12-02T07:22:28.837 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90325.log 2023-12-02T07:22:28.838 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113165.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.113165.log.gz 2023-12-02T07:22:28.846 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101306.log 2023-12-02T07:22:28.847 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.90325.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90325.log.gz 2023-12-02T07:22:28.855 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112725.log 2023-12-02T07:22:28.856 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101306.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101306.log.gz 2023-12-02T07:22:28.870 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91879.log 2023-12-02T07:22:28.870 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112725.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.112725.log.gz 2023-12-02T07:22:28.875 INFO:teuthology.orchestra.run.smithi057.stderr: 94.6% -- replaced with /var/log/ceph/cluster1-client.mirror.2.91691.log.gz 2023-12-02T07:22:28.888 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109410.log 2023-12-02T07:22:28.889 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.91879.log: 80.6% -- replaced with /var/log/ceph/cluster1-client.mirror.91879.log.gz 2023-12-02T07:22:28.911 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115085.log 2023-12-02T07:22:28.911 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.109410.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.109410.log.gz 2023-12-02T07:22:28.912 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.105103.log.gz 2023-12-02T07:22:28.913 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105964.log 2023-12-02T07:22:28.913 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115085.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115085.log.gz 2023-12-02T07:22:28.914 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95481.log 2023-12-02T07:22:28.915 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.105964.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.105964.log.gz 2023-12-02T07:22:28.916 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115105.log 2023-12-02T07:22:28.921 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.95481.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95481.log.gz 2023-12-02T07:22:28.922 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113385.log 2023-12-02T07:22:28.922 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115105.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.115105.log.gz 2023-12-02T07:22:28.924 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106028.log 2023-12-02T07:22:28.924 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113385.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.113385.log.gz 2023-12-02T07:22:28.924 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98608.log 2023-12-02T07:22:28.925 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.106028.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.106028.log.gz 2023-12-02T07:22:28.925 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107505.log 2023-12-02T07:22:28.925 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.98608.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.98608.log.gz 2023-12-02T07:22:28.926 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97880.log 2023-12-02T07:22:28.926 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.107505.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.107505.log.gz 2023-12-02T07:22:28.927 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99392.log 2023-12-02T07:22:28.936 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.97880.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.97880.log.gz 2023-12-02T07:22:28.937 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96478.log 2023-12-02T07:22:28.943 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.99392.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99392.log.gz 2023-12-02T07:22:28.943 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108601.log 2023-12-02T07:22:28.949 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.96478.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.96478.log.gz 2023-12-02T07:22:28.949 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94938.log 2023-12-02T07:22:28.950 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.108601.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.108601.log.gz 2023-12-02T07:22:28.951 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109306.log 2023-12-02T07:22:28.956 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.94938.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.94938.log.gz 2023-12-02T07:22:28.957 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.106390.log 2023-12-02T07:22:28.958 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.109306.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.109306.log.gz 2023-12-02T07:22:28.959 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113825.log 2023-12-02T07:22:28.972 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.0.106390.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98896.log 2023-12-02T07:22:28.973 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113825.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113825.log.gz 2023-12-02T07:22:28.986 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94675.log 2023-12-02T07:22:28.987 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.98896.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.98896.log.gz 2023-12-02T07:22:28.988 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111007.log 2023-12-02T07:22:28.995 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.94675.log: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99587.log 2023-12-02T07:22:28.995 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.94675.log.gz 2023-12-02T07:22:28.996 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.111007.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.0.106390.log.gz 2023-12-02T07:22:28.999 INFO:teuthology.orchestra.run.smithi057.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.111007.log.gz 2023-12-02T07:22:28.999 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111533.log 2023-12-02T07:22:29.004 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.99587.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.99587.log.gz 2023-12-02T07:22:29.005 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92205.log 2023-12-02T07:22:29.006 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.111533.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.111533.log.gz 2023-12-02T07:22:29.008 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114425.log 2023-12-02T07:22:29.008 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.92205.log: 80.2% -- replaced with /var/log/ceph/cluster1-client.admin.92205.log.gz 2023-12-02T07:22:29.008 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103629.log 2023-12-02T07:22:29.010 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114425.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114425.log.gz 2023-12-02T07:22:29.010 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97102.log 2023-12-02T07:22:29.011 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103629.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103629.log.gz 2023-12-02T07:22:29.011 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96768.log 2023-12-02T07:22:29.012 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.97102.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.97102.log.gz 2023-12-02T07:22:29.013 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95934.log 2023-12-02T07:22:29.014 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.96768.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.96768.log.gz 2023-12-02T07:22:29.014 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116205.log 2023-12-02T07:22:29.015 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.95934.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95934.log.gz 2023-12-02T07:22:29.015 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94063.log 2023-12-02T07:22:29.017 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116205.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116205.log.gz 2023-12-02T07:22:29.017 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110721.log 2023-12-02T07:22:29.018 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.94063.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102982.log.gz 2023-12-02T07:22:29.018 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.94063.log.gz 2023-12-02T07:22:29.018 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92273.log 2023-12-02T07:22:29.018 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.110721.log: --verbose -- /var/log/ceph/cluster1-client.mirror.115185.log 2023-12-02T07:22:29.021 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.92273.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110721.log.gz 2023-12-02T07:22:29.021 INFO:teuthology.orchestra.run.smithi057.stderr: 4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.92273.log.gz 2023-12-02T07:22:29.021 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114285.log 2023-12-02T07:22:29.021 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115185.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102107.log 2023-12-02T07:22:29.022 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115185.log.gz 2023-12-02T07:22:29.022 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114285.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96145.log 2023-12-02T07:22:29.022 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102107.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114285.log.gz 2023-12-02T07:22:29.022 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95975.log 2023-12-02T07:22:29.023 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.96145.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.96145.log.gz 2023-12-02T07:22:29.023 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95629.log 2023-12-02T07:22:29.029 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.95975.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.95975.log.gz 2023-12-02T07:22:29.029 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114025.log 2023-12-02T07:22:29.030 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.95629.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95629.log.gz 2023-12-02T07:22:29.030 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102255.log 2023-12-02T07:22:29.032 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114025.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114025.log.gz 2023-12-02T07:22:29.032 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110790.log 2023-12-02T07:22:29.045 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102255.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107746.log 2023-12-02T07:22:29.047 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.110790.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110790.log.gz 2023-12-02T07:22:29.052 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92751.log 2023-12-02T07:22:29.063 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.107746.log: 93.7% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107746.log.gz 2023-12-02T07:22:29.063 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.102107.log.gz 2023-12-02T07:22:29.065 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98872.log 2023-12-02T07:22:29.065 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.92751.log: 4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.92751.log.gz 2023-12-02T07:22:29.065 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112070.log 2023-12-02T07:22:29.065 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.98872.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98872.log.gz 2023-12-02T07:22:29.067 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108916.log 2023-12-02T07:22:29.067 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112070.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.112070.log.gz 2023-12-02T07:22:29.067 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91656.log 2023-12-02T07:22:29.068 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.108916.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108916.log.gz 2023-12-02T07:22:29.069 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99091.log 2023-12-02T07:22:29.069 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.91656.log: 74.8% -- replaced with /var/log/ceph/cluster2-client.mirror.91656.log.gz 2023-12-02T07:22:29.070 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110342.log 2023-12-02T07:22:29.075 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.99091.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99091.log.gz 2023-12-02T07:22:29.076 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108442.log 2023-12-02T07:22:29.077 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.110342.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.110342.log.gz 2023-12-02T07:22:29.077 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104469.log 2023-12-02T07:22:29.078 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.108442.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108442.log.gz 2023-12-02T07:22:29.079 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.98405.log 2023-12-02T07:22:29.080 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104469.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.104469.log.gz 2023-12-02T07:22:29.080 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103089.log 2023-12-02T07:22:29.081 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.0.98405.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101819.log.gz 2023-12-02T07:22:29.083 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114485.log 2023-12-02T07:22:29.083 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103089.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103089.log.gz 2023-12-02T07:22:29.084 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104491.log 2023-12-02T07:22:29.084 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114485.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114485.log.gz 2023-12-02T07:22:29.086 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102213.log 2023-12-02T07:22:29.086 INFO:teuthology.orchestra.run.smithi057.stderr: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.0.98405.log.gz 2023-12-02T07:22:29.086 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104491.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111886.log 2023-12-02T07:22:29.086 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102213.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102213.log.gz 2023-12-02T07:22:29.087 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104136.log 2023-12-02T07:22:29.089 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.111886.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.111886.log.gz 2023-12-02T07:22:29.089 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94777.log 2023-12-02T07:22:29.102 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104136.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112485.log 2023-12-02T07:22:29.103 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.94777.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.94777.log.gz 2023-12-02T07:22:29.116 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108988.log 2023-12-02T07:22:29.116 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112485.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.112485.log.gz 2023-12-02T07:22:29.127 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109265.log 2023-12-02T07:22:29.128 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.108988.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108988.log.gz 2023-12-02T07:22:29.140 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110320.log 2023-12-02T07:22:29.140 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.109265.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.109265.log.gz 2023-12-02T07:22:29.151 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103972.log.gz 2023-12-02T07:22:29.151 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110524.log/var/log/ceph/cluster1-client.mirror.110320.log: 2023-12-02T07:22:29.153 INFO:teuthology.orchestra.run.smithi057.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110320.log.gz 2023-12-02T07:22:29.153 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92256.log 2023-12-02T07:22:29.155 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.110524.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.110524.log.gz 2023-12-02T07:22:29.156 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103444.log 2023-12-02T07:22:29.156 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.92256.log: 4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.92256.log.gz 2023-12-02T07:22:29.156 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107770.log 2023-12-02T07:22:29.159 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103444.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103444.log.gz 2023-12-02T07:22:29.159 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115739.log 2023-12-02T07:22:29.159 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.107770.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.107770.log.gz 2023-12-02T07:22:29.159 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114445.log 2023-12-02T07:22:29.159 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115739.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115739.log.gz 2023-12-02T07:22:29.160 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94893.log 2023-12-02T07:22:29.160 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114445.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114445.log.gz 2023-12-02T07:22:29.160 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98848.log 2023-12-02T07:22:29.161 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.94893.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.94893.log.gz 2023-12-02T07:22:29.161 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108289.log 2023-12-02T07:22:29.161 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.98848.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.98848.log.gz 2023-12-02T07:22:29.162 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85737.log 2023-12-02T07:22:29.163 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.108289.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.108289.log.gz 2023-12-02T07:22:29.164 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107818.log 2023-12-02T07:22:29.165 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.85737.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85737.log.gz 2023-12-02T07:22:29.165 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107075.log 2023-12-02T07:22:29.166 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.107818.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.107818.log.gz 2023-12-02T07:22:29.166 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100780.log 2023-12-02T07:22:29.168 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.107075.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.107075.log.gz 2023-12-02T07:22:29.168 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106185.log 2023-12-02T07:22:29.169 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100780.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100780.log.gz 2023-12-02T07:22:29.169 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108538.log 2023-12-02T07:22:29.171 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.106185.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.106185.log.gz 2023-12-02T07:22:29.172 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113545.log 2023-12-02T07:22:29.173 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.108538.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.108538.log.gz 2023-12-02T07:22:29.173 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92358.log 2023-12-02T07:22:29.175 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113545.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.113545.log.gz 2023-12-02T07:22:29.175 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90753.log 2023-12-02T07:22:29.177 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.92358.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.92358.log.gz 2023-12-02T07:22:29.177 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103930.log 2023-12-02T07:22:29.177 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.90753.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90753.log.gz 2023-12-02T07:22:29.177 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103592.log 2023-12-02T07:22:29.179 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103930.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103930.log.gz 2023-12-02T07:22:29.179 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96089.log 2023-12-02T07:22:29.180 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103592.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103592.log.gz 2023-12-02T07:22:29.180 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106846.log 2023-12-02T07:22:29.181 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.96089.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.96089.log.gz 2023-12-02T07:22:29.182 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112425.log 2023-12-02T07:22:29.183 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.106846.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.106846.log.gz 2023-12-02T07:22:29.183 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109033.log 2023-12-02T07:22:29.184 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112425.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112425.log.gz 2023-12-02T07:22:29.184 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99693.log 2023-12-02T07:22:29.187 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.109033.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.109033.log.gz 2023-12-02T07:22:29.187 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105574.log 2023-12-02T07:22:29.193 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.99693.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99693.log.gz 2023-12-02T07:22:29.193 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104077.log 2023-12-02T07:22:29.193 INFO:teuthology.orchestra.run.smithi057.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.94651.log.gz 2023-12-02T07:22:29.198 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105574.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94869.log 2023-12-02T07:22:29.198 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104077.log: 88.6% 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.105574.log.gz 2023-12-02T07:22:29.198 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.104077.log.gz 2023-12-02T07:22:29.198 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105465.log 2023-12-02T07:22:29.198 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.94869.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.93611.log 2023-12-02T07:22:29.198 INFO:teuthology.orchestra.run.smithi057.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.94869.log.gz 2023-12-02T07:22:29.199 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105465.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91919.log 2023-12-02T07:22:29.199 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.93611.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.105465.log.gz 2023-12-02T07:22:29.199 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97167.log 2023-12-02T07:22:29.199 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.93611.log.gz 2023-12-02T07:22:29.199 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.91919.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94705.log 2023-12-02T07:22:29.199 INFO:teuthology.orchestra.run.smithi057.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.91919.log.gz 2023-12-02T07:22:29.199 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.97167.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98985.log 2023-12-02T07:22:29.200 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.94705.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.94705.log.gz 2023-12-02T07:22:29.200 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100082.log 2023-12-02T07:22:29.204 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.98985.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.97167.log.gz 2023-12-02T07:22:29.204 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95538.log 2023-12-02T07:22:29.204 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100082.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.100082.log.gz 2023-12-02T07:22:29.209 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105803.log 2023-12-02T07:22:29.209 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.95538.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.98985.log.gz 2023-12-02T07:22:29.209 INFO:teuthology.orchestra.run.smithi057.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.95538.log.gz 2023-12-02T07:22:29.209 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105659.log 2023-12-02T07:22:29.209 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105803.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115812.log 2023-12-02T07:22:29.209 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.105803.log.gz 2023-12-02T07:22:29.210 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105659.log: 2.6% -- replaced with /var/log/ceph/cluster2-client.mirror.105659.log.gz 2023-12-02T07:22:29.210 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104866.log 2023-12-02T07:22:29.210 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115812.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115812.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104528.log 2023-12-02T07:22:29.210 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-12-02T07:22:29.210 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90423.log 2023-12-02T07:22:29.210 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104866.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.104866.log.gz 2023-12-02T07:22:29.210 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104528.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98697.log 2023-12-02T07:22:29.210 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.90423.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90423.log.gz 2023-12-02T07:22:29.211 INFO:teuthology.orchestra.run.smithi057.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104528.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103227.log 2023-12-02T07:22:29.212 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-12-02T07:22:29.212 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.98697.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109523.log 2023-12-02T07:22:29.215 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103227.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98697.log.gz 2023-12-02T07:22:29.215 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115325.log 2023-12-02T07:22:29.218 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.109523.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.109523.log.gz 2023-12-02T07:22:29.219 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113885.log 2023-12-02T07:22:29.220 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115325.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.115325.log.gz 2023-12-02T07:22:29.220 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116348.log 2023-12-02T07:22:29.220 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113885.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113885.log.gz 2023-12-02T07:22:29.223 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115285.log 2023-12-02T07:22:29.223 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116348.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.mirror.116348.log.gz 2023-12-02T07:22:29.223 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110769.log 2023-12-02T07:22:29.223 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115285.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.115285.log.gz 2023-12-02T07:22:29.223 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97476.log 2023-12-02T07:22:29.225 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.110769.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.110769.log.gz 2023-12-02T07:22:29.225 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96235.log 2023-12-02T07:22:29.226 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.97476.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.97476.log.gz 2023-12-02T07:22:29.226 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.93651.log 2023-12-02T07:22:29.227 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.96235.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.96235.log.gz 2023-12-02T07:22:29.227 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111248.log 2023-12-02T07:22:29.237 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.2.93651.log: 93.1%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95256.log 2023-12-02T07:22:29.240 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.111248.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.2.93651.log.gz 2023-12-02T07:22:29.240 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.111248.log.gz 2023-12-02T07:22:29.240 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104570.log 2023-12-02T07:22:29.243 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.95256.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95256.log.gz 2023-12-02T07:22:29.243 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109482.log 2023-12-02T07:22:29.256 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104570.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94417.log 2023-12-02T07:22:29.257 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.109482.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.109482.log.gz 2023-12-02T07:22:29.263 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104801.log 2023-12-02T07:22:29.276 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.94417.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.116470.log 2023-12-02T07:22:29.276 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.94417.log.gz 2023-12-02T07:22:29.277 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104801.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104801.log.gz 2023-12-02T07:22:29.290 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90465.log 2023-12-02T07:22:29.290 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.116470.log: 6.1% -- replaced with /var/log/ceph/cluster1-client.admin.116470.log.gz 2023-12-02T07:22:29.303 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98221.log 2023-12-02T07:22:29.303 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.90465.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90465.log.gz 2023-12-02T07:22:29.312 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99263.log 2023-12-02T07:22:29.318 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.98221.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101907.log.gz 2023-12-02T07:22:29.319 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109286.log 2023-12-02T07:22:29.325 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.99263.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99263.log.gz 2023-12-02T07:22:29.325 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100102.log 2023-12-02T07:22:29.328 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.109286.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.109286.log.gz 2023-12-02T07:22:29.328 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103385.log 2023-12-02T07:22:29.328 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100102.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100102.log.gz 2023-12-02T07:22:29.329 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96458.log 2023-12-02T07:22:29.330 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103385.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.98221.log.gz 2023-12-02T07:22:29.347 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100822.log 2023-12-02T07:22:29.353 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.96458.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.96458.log.gz 2023-12-02T07:22:29.366 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103834.log 2023-12-02T07:22:29.380 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100822.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115829.log 2023-12-02T07:22:29.381 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103834.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103834.log.gz 2023-12-02T07:22:29.390 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108872.log 2023-12-02T07:22:29.391 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115829.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115829.log.gz 2023-12-02T07:22:29.398 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97428.log 2023-12-02T07:22:29.405 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.108872.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.108872.log.gz 2023-12-02T07:22:29.410 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99005.log 2023-12-02T07:22:29.411 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.97428.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.97428.log.gz 2023-12-02T07:22:29.421 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110231.log 2023-12-02T07:22:29.435 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.99005.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112046.log 2023-12-02T07:22:29.435 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.110231.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.110231.log.gz 2023-12-02T07:22:29.439 INFO:teuthology.orchestra.run.smithi057.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99005.log.gz 2023-12-02T07:22:29.446 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114845.log 2023-12-02T07:22:29.447 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112046.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.112046.log.gz 2023-12-02T07:22:29.455 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113345.log 2023-12-02T07:22:29.464 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114845.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112705.log 2023-12-02T07:22:29.468 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113345.log: 87.4% 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.114845.log.gz 2023-12-02T07:22:29.473 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94917.log 2023-12-02T07:22:29.476 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112705.log: -- replaced with /var/log/ceph/cluster1-client.mirror.113345.log.gz 2023-12-02T07:22:29.486 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93877.log 2023-12-02T07:22:29.486 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.112705.log.gz 2023-12-02T07:22:29.487 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.94917.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94917.log.gz 2023-12-02T07:22:29.500 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113125.log 2023-12-02T07:22:29.501 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.93877.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.93877.log.gz 2023-12-02T07:22:29.514 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97210.log 2023-12-02T07:22:29.522 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113125.log: 87.9% 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104136.log.gz 2023-12-02T07:22:29.528 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.113125.log.gz 2023-12-02T07:22:29.535 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101406.log 2023-12-02T07:22:29.541 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.97210.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.97210.log.gz 2023-12-02T07:22:29.545 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93851.log 2023-12-02T07:22:29.559 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101406.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101406.log.gz 2023-12-02T07:22:29.564 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99564.log 2023-12-02T07:22:29.565 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.93851.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.93851.log.gz 2023-12-02T07:22:29.578 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107964.log 2023-12-02T07:22:29.593 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.99564.log: 93.4% 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104491.log.gz 2023-12-02T07:22:29.595 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106048.log 2023-12-02T07:22:29.595 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.107964.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.107964.log.gz 2023-12-02T07:22:29.595 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94084.log 2023-12-02T07:22:29.596 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.106048.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.106048.log.gz 2023-12-02T07:22:29.596 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116319.log 2023-12-02T07:22:29.597 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.94084.log: -- replaced with /var/log/ceph/cluster1-client.mirror.99564.log.gz 2023-12-02T07:22:29.601 INFO:teuthology.orchestra.run.smithi057.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.94084.log.gz 2023-12-02T07:22:29.602 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110810.log 2023-12-02T07:22:29.603 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.116319.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.116319.log.gz 2023-12-02T07:22:29.603 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.112228.log 2023-12-02T07:22:29.608 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.110810.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110810.log.gz 2023-12-02T07:22:29.609 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.112286.log 2023-12-02T07:22:29.609 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.1.112228.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.1.112228.log.gz 2023-12-02T07:22:29.609 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115461.log 2023-12-02T07:22:29.609 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.112286.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.112286.log.gz 2023-12-02T07:22:29.609 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97232.log 2023-12-02T07:22:29.609 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.115461.log: 70.1% -- replaced with /var/log/ceph/cluster1-client.admin.115461.log.gz 2023-12-02T07:22:29.609 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99521.log 2023-12-02T07:22:29.610 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.97232.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.97232.log.gz 2023-12-02T07:22:29.611 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100223.log 2023-12-02T07:22:29.616 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.99521.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99521.log.gz 2023-12-02T07:22:29.616 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101326.log 2023-12-02T07:22:29.617 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100223.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.100223.log.gz 2023-12-02T07:22:29.618 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104374.log 2023-12-02T07:22:29.620 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101326.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101326.log.gz 2023-12-02T07:22:29.620 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91263.log 2023-12-02T07:22:29.626 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104374.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102255.log.gz 2023-12-02T07:22:29.629 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116412.log 2023-12-02T07:22:29.629 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.91263.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.91263.log.gz 2023-12-02T07:22:29.629 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114385.log 2023-12-02T07:22:29.629 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116412.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.mirror.116412.log.gz 2023-12-02T07:22:29.629 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104925.log 2023-12-02T07:22:29.629 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114385.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.114385.log.gz 2023-12-02T07:22:29.630 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101991.log 2023-12-02T07:22:29.642 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104925.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96820.log 2023-12-02T07:22:29.645 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101991.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103227.log.gz 2023-12-02T07:22:29.655 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112885.log 2023-12-02T07:22:29.656 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.96820.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.96820.log.gz 2023-12-02T07:22:29.668 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115406.log 2023-12-02T07:22:29.669 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112885.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.112885.log.gz 2023-12-02T07:22:29.688 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100022.log 2023-12-02T07:22:29.688 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115406.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115406.log.gz 2023-12-02T07:22:29.696 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106538.log 2023-12-02T07:22:29.709 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100022.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.100022.log.gz 2023-12-02T07:22:29.714 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105594.log 2023-12-02T07:22:29.715 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.106538.log: 95.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106538.log.gz 2023-12-02T07:22:29.727 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103651.log 2023-12-02T07:22:29.729 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105594.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.105594.log.gz 2023-12-02T07:22:29.744 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104156.log 2023-12-02T07:22:29.758 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103651.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110002.log 2023-12-02T07:22:29.758 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104156.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104156.log.gz 2023-12-02T07:22:29.758 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104570.log.gz 2023-12-02T07:22:29.760 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93587.log 2023-12-02T07:22:29.760 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.110002.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110002.log.gz 2023-12-02T07:22:29.760 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112925.log 2023-12-02T07:22:29.776 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.93587.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.91815.log 2023-12-02T07:22:29.776 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112925.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.112925.log.gz 2023-12-02T07:22:29.795 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104009.log 2023-12-02T07:22:29.807 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.0.91815.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101081.log 2023-12-02T07:22:29.807 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104009.log: 92.2% 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104009.log.gz 2023-12-02T07:22:29.818 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104784.log 2023-12-02T07:22:29.820 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101081.log: -- replaced with /var/log/ceph/cluster2-client.mirror.0.91815.log.gz 2023-12-02T07:22:29.834 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114345.log 2023-12-02T07:22:29.835 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104784.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104784.log.gz 2023-12-02T07:22:29.847 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98677.log 2023-12-02T07:22:29.848 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114345.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.114345.log.gz 2023-12-02T07:22:29.862 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110938.log 2023-12-02T07:22:29.868 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.98677.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98677.log.gz 2023-12-02T07:22:29.881 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96501.log 2023-12-02T07:22:29.882 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.110938.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110938.log.gz 2023-12-02T07:22:29.896 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110207.log 2023-12-02T07:22:29.901 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.96501.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.96501.log.gz 2023-12-02T07:22:29.908 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95838.log 2023-12-02T07:22:29.909 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.110207.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.110207.log.gz 2023-12-02T07:22:29.918 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101059.log 2023-12-02T07:22:29.918 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.95838.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.95838.log.gz 2023-12-02T07:22:29.926 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85576.log 2023-12-02T07:22:29.928 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101059.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101059.log.gz 2023-12-02T07:22:29.945 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95651.log 2023-12-02T07:22:29.946 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.85576.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85576.log.gz 2023-12-02T07:22:29.958 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116022.log 2023-12-02T07:22:29.972 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.95651.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101848.log 2023-12-02T07:22:29.972 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116022.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116022.log.gz 2023-12-02T07:22:29.980 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100002.log 2023-12-02T07:22:29.981 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101848.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101848.log.gz 2023-12-02T07:22:29.993 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101797.log 2023-12-02T07:22:30.000 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100002.log: 87.9% 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100822.log.gz 2023-12-02T07:22:30.005 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.100002.log.gz 2023-12-02T07:22:30.005 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113085.log 2023-12-02T07:22:30.005 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101797.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101797.log.gz 2023-12-02T07:22:30.017 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105372.log 2023-12-02T07:22:30.018 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113085.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.113085.log.gz 2023-12-02T07:22:30.036 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105722.log 2023-12-02T07:22:30.036 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105372.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.105372.log.gz 2023-12-02T07:22:30.042 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104374.log.gz 2023-12-02T07:22:30.050 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105251.log 2023-12-02T07:22:30.051 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.105722.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.105722.log.gz 2023-12-02T07:22:30.069 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.93795.log 2023-12-02T07:22:30.069 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105251.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.105251.log.gz 2023-12-02T07:22:30.080 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102612.log 2023-12-02T07:22:30.080 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.93795.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.93795.log.gz 2023-12-02T07:22:30.088 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91347.log 2023-12-02T07:22:30.104 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102612.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103385.log.gz 2023-12-02T07:22:30.105 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.96602.log 2023-12-02T07:22:30.105 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.91347.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.91347.log.gz 2023-12-02T07:22:30.116 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114925.log 2023-12-02T07:22:30.128 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.2.96602.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104511.log 2023-12-02T07:22:30.128 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.114925.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.114925.log.gz 2023-12-02T07:22:30.136 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103730.log 2023-12-02T07:22:30.145 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104511.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103168.log 2023-12-02T07:22:30.149 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103730.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.104511.log.gz 2023-12-02T07:22:30.162 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102011.log 2023-12-02T07:22:30.163 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103168.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103168.log.gz 2023-12-02T07:22:30.177 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115496.log 2023-12-02T07:22:30.179 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102011.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.102011.log.gz 2023-12-02T07:22:30.192 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104925.log.gz 2023-12-02T07:22:30.197 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107276.log 2023-12-02T07:22:30.197 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115496.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115496.log.gz 2023-12-02T07:22:30.215 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108310.log 2023-12-02T07:22:30.216 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.107276.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.107276.log.gz 2023-12-02T07:22:30.228 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103326.log 2023-12-02T07:22:30.230 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.108310.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.108310.log.gz 2023-12-02T07:22:30.241 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112405.log 2023-12-02T07:22:30.241 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103326.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103326.log.gz 2023-12-02T07:22:30.251 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99177.log 2023-12-02T07:22:30.253 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112405.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.112405.log.gz 2023-12-02T07:22:30.263 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115305.log 2023-12-02T07:22:30.272 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.99177.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107123.log 2023-12-02T07:22:30.272 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.99177.log.gz 2023-12-02T07:22:30.277 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.115305.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101991.log.gz 2023-12-02T07:22:30.280 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.115305.log.gz 2023-12-02T07:22:30.284 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116246.log 2023-12-02T07:22:30.288 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.107123.log: 87.4% 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.2.96602.log.gz 2023-12-02T07:22:30.288 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.107123.log.gz 2023-12-02T07:22:30.290 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106959.log 2023-12-02T07:22:30.290 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.116246.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116246.log.gz 2023-12-02T07:22:30.291 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108514.log 2023-12-02T07:22:30.292 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.106959.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.106959.log.gz 2023-12-02T07:22:30.293 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110279.log 2023-12-02T07:22:30.295 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.108514.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.108514.log.gz 2023-12-02T07:22:30.295 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96846.log 2023-12-02T07:22:30.296 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.110279.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.110279.log.gz 2023-12-02T07:22:30.296 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105172.log 2023-12-02T07:22:30.297 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.96846.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.96846.log.gz 2023-12-02T07:22:30.297 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113265.log 2023-12-02T07:22:30.299 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105172.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.105172.log.gz 2023-12-02T07:22:30.299 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102797.log 2023-12-02T07:22:30.300 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113265.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.113265.log.gz 2023-12-02T07:22:30.300 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108695.log 2023-12-02T07:22:30.319 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102797.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111495.log 2023-12-02T07:22:30.320 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.108695.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.108695.log.gz 2023-12-02T07:22:30.333 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103424.log 2023-12-02T07:22:30.334 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.111495.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.111495.log.gz 2023-12-02T07:22:30.348 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115905.log 2023-12-02T07:22:30.348 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.103424.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103424.log.gz 2023-12-02T07:22:30.364 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113485.log 2023-12-02T07:22:30.364 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.115905.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.admin.115905.log.gz 2023-12-02T07:22:30.379 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108193.log 2023-12-02T07:22:30.380 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113485.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.113485.log.gz 2023-12-02T07:22:30.390 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105745.log 2023-12-02T07:22:30.390 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.108193.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108193.log.gz 2023-12-02T07:22:30.394 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103651.log.gz 2023-12-02T07:22:30.403 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101081.log.gz 2023-12-02T07:22:30.408 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102511.log 2023-12-02T07:22:30.409 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.105745.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101681.log 2023-12-02T07:22:30.409 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102511.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.105745.log.gz 2023-12-02T07:22:30.410 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102511.log.gz 2023-12-02T07:22:30.411 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104235.log 2023-12-02T07:22:30.412 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101681.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101681.log.gz 2023-12-02T07:22:30.412 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113525.log 2023-12-02T07:22:30.414 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104235.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104235.log.gz 2023-12-02T07:22:30.414 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97589.log 2023-12-02T07:22:30.414 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103730.log.gz 2023-12-02T07:22:30.414 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113525.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113525.log.gz 2023-12-02T07:22:30.415 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113245.log 2023-12-02T07:22:30.415 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.97589.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2023-12-02T07:22:30.417 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113245.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113245.log.gz 2023-12-02T07:22:30.418 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.93587.log.gz 2023-12-02T07:22:30.418 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91008.log 2023-12-02T07:22:30.418 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104193.log 2023-12-02T07:22:30.418 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.admin.91008.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96040.log 2023-12-02T07:22:30.418 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.91008.log.gz 2023-12-02T07:22:30.418 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104193.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102755.log 2023-12-02T07:22:30.418 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.96040.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.104193.log.gz 2023-12-02T07:22:30.420 INFO:teuthology.orchestra.run.smithi057.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.96040.log.gz 2023-12-02T07:22:30.420 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107920.log 2023-12-02T07:22:30.420 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102755.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102755.log.gz 2023-12-02T07:22:30.421 INFO:teuthology.orchestra.run.smithi057.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.97589.log.gz 2023-12-02T07:22:30.421 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103893.log 2023-12-02T07:22:30.421 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94203.log 2023-12-02T07:22:30.422 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.107920.log: /var/log/ceph/cluster2-client.mirror.103893.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.107920.log.gz 2023-12-02T07:22:30.422 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110587.log 2023-12-02T07:22:30.422 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.94203.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.94203.log.gz 2023-12-02T07:22:30.422 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113405.log 2023-12-02T07:22:30.423 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.110587.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110587.log.gz 2023-12-02T07:22:30.423 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112965.log 2023-12-02T07:22:30.424 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113405.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113405.log.gz 2023-12-02T07:22:30.425 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91095.log 2023-12-02T07:22:30.425 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110300.log 2023-12-02T07:22:30.426 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112965.log: /var/log/ceph/cluster2-client.admin.91095.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.91095.log.gz 2023-12-02T07:22:30.426 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112965.log.gz 2023-12-02T07:22:30.427 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108217.log 2023-12-02T07:22:30.427 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109672.log 2023-12-02T07:22:30.427 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.108217.log: /var/log/ceph/cluster2-client.mirror.110300.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.108217.log.gz 2023-12-02T07:22:30.428 INFO:teuthology.orchestra.run.smithi057.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.110300.log.gz 2023-12-02T07:22:30.429 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97278.log 2023-12-02T07:22:30.429 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.109672.log: 94.6% -- replaced with /var/log/ceph/cluster1-client.mirror.109672.log.gz 2023-12-02T07:22:30.430 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102834.log 2023-12-02T07:22:30.431 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.admin.97278.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.97278.log.gz 2023-12-02T07:22:30.432 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100333.log 2023-12-02T07:22:30.432 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102834.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102834.log.gz 2023-12-02T07:22:30.433 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94821.log 2023-12-02T07:22:30.433 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.100333.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100333.log.gz 2023-12-02T07:22:30.433 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112985.log 2023-12-02T07:22:30.434 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.94821.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94821.log.gz 2023-12-02T07:22:30.436 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101022.log 2023-12-02T07:22:30.437 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102612.log.gz 2023-12-02T07:22:30.437 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.112985.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.112985.log.gz 2023-12-02T07:22:30.437 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101760.log 2023-12-02T07:22:30.437 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101022.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108743.log 2023-12-02T07:22:30.437 INFO:teuthology.orchestra.run.smithi057.stderr: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.101022.log.gz 2023-12-02T07:22:30.437 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101760.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.101760.log.gz 2023-12-02T07:22:30.438 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105899.log 2023-12-02T07:22:30.439 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101505.log 2023-12-02T07:22:30.439 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.108743.log: /var/log/ceph/cluster1-client.mirror.105899.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108743.log.gz 2023-12-02T07:22:30.439 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102300.log 2023-12-02T07:22:30.439 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.105899.log.gz 2023-12-02T07:22:30.440 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.101505.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.101505.log.gz 2023-12-02T07:22:30.440 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116482.log 2023-12-02T07:22:30.440 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95369.log 2023-12-02T07:22:30.441 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.102300.log: /var/log/ceph/cluster2-client.admin.116482.log: 10.2% -- replaced with /var/log/ceph/cluster2-client.admin.116482.log.gz 2023-12-02T07:22:30.441 INFO:teuthology.orchestra.run.smithi057.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102300.log.gz 2023-12-02T07:22:30.442 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112545.log 2023-12-02T07:22:30.442 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104114.log 2023-12-02T07:22:30.443 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.95369.log: /var/log/ceph/cluster1-client.mirror.112545.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.95369.log.gz 2023-12-02T07:22:30.443 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.112545.log.gz 2023-12-02T07:22:30.444 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107210.log 2023-12-02T07:22:30.444 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113765.log 2023-12-02T07:22:30.444 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster2-client.mirror.104114.log: /var/log/ceph/cluster2-client.mirror.107210.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.104114.log.gz 2023-12-02T07:22:30.445 INFO:teuthology.orchestra.run.smithi057.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.107210.log.gz 2023-12-02T07:22:30.445 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cluster1-client.mirror.113765.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.113765.log.gz 2023-12-02T07:22:30.471 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.95651.log.gz 2023-12-02T07:22:30.615 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102797.log.gz 2023-12-02T07:22:30.705 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103893.log.gz 2023-12-02T07:22:30.882 INFO:teuthology.orchestra.run.smithi029.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2023-12-02T07:22:31.264 INFO:teuthology.orchestra.run.smithi029.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2023-12-02T07:22:31.407 INFO:teuthology.orchestra.run.smithi029.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2023-12-02T07:22:31.409 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-12-02T07:22:31.409 INFO:teuthology.orchestra.run.smithi029.stderr:real 0m5.988s 2023-12-02T07:22:31.409 INFO:teuthology.orchestra.run.smithi029.stderr:user 0m17.805s 2023-12-02T07:22:31.410 INFO:teuthology.orchestra.run.smithi029.stderr:sys 0m0.648s 2023-12-02T07:23:24.580 INFO:teuthology.orchestra.run.smithi057.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2023-12-02T07:23:28.248 INFO:teuthology.orchestra.run.smithi057.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2023-12-02T07:23:33.830 INFO:teuthology.orchestra.run.smithi057.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2023-12-02T07:23:33.832 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-12-02T07:23:33.832 INFO:teuthology.orchestra.run.smithi057.stderr:real 1m8.384s 2023-12-02T07:23:33.832 INFO:teuthology.orchestra.run.smithi057.stderr:user 3m26.501s 2023-12-02T07:23:33.832 INFO:teuthology.orchestra.run.smithi057.stderr:sys 0m7.570s 2023-12-02T07:23:33.833 INFO:tasks.ceph:Archiving logs... 2023-12-02T07:23:33.833 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-01_18:01:25-rbd-main-distro-default-smithi/7474803/remote/smithi029/log 2023-12-02T07:23:33.834 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-02T07:23:36.308 DEBUG:teuthology.misc:Transferring archived files from smithi057:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-01_18:01:25-rbd-main-distro-default-smithi/7474803/remote/smithi057/log 2023-12-02T07:23:36.309 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-02T07:24:06.053 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi029 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json' 2023-12-02T07:24:06.058 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-02T07:24:06.076 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing 2023-12-02T07:24:06.084 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-02T07:24:06.084 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-02T07:24:06.107 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-02T07:24:06.195 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-02T07:24:06.213 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-02T07:24:06.213 DEBUG:teuthology.orchestra.run.smithi029:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-02T07:24:06.216 DEBUG:teuthology.orchestra.run.smithi057:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-02T07:24:06.231 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2023-12-02T07:24:06.233 INFO:teuthology.orchestra.run.smithi057.stderr:bash: line 1: ntpq: command not found 2023-12-02T07:24:06.235 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-02T07:24:06.235 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2023-12-02T07:24:06.236 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 41 +849us[ +762us] +/- 101ms 2023-12-02T07:24:06.236 INFO:teuthology.orchestra.run.smithi029.stdout:^* hv02.front.sepia.ceph.com 3 7 377 37 -2749us[-2836us] +/- 64ms 2023-12-02T07:24:06.236 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 39 +2932us[+2845us] +/- 72ms 2023-12-02T07:24:06.236 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-12-02T07:24:06.288 INFO:teuthology.orchestra.run.smithi057.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-02T07:24:06.288 INFO:teuthology.orchestra.run.smithi057.stdout:=============================================================================== 2023-12-02T07:24:06.289 INFO:teuthology.orchestra.run.smithi057.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 102 +792us[ +736us] +/- 100ms 2023-12-02T07:24:06.289 INFO:teuthology.orchestra.run.smithi057.stdout:^* hv02.front.sepia.ceph.com 3 6 377 40 -2852us[-2909us] +/- 64ms 2023-12-02T07:24:06.289 INFO:teuthology.orchestra.run.smithi057.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 37 +2868us[+2868us] +/- 72ms 2023-12-02T07:24:06.289 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-12-02T07:24:06.289 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-02T07:24:06.304 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-02T07:24:06.305 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-02T07:24:06.347 DEBUG:teuthology.orchestra.run.smithi029:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-02T07:24:06.458 DEBUG:teuthology.orchestra.run.smithi057:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-02T07:24:06.661 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-02T07:24:06.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-02T07:24:06.700 DEBUG:teuthology.orchestra.run.smithi057:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-02T07:24:06.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-02T07:24:06.739 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-02T07:24:06.754 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-02T07:24:06.806 INFO:teuthology.task.internal:Duration was 17229.490659 seconds 2023-12-02T07:24:06.807 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-02T07:24:06.855 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-02T07:24:06.855 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-02T07:24:06.858 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-02T07:24:06.911 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-02T07:24:06.938 INFO:teuthology.orchestra.run.smithi057.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-02T07:24:07.334 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-02T07:24:07.335 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi029.front.sepia.ceph.com 2023-12-02T07:24:07.335 DEBUG:teuthology.orchestra.run.smithi029:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-02T07:24:07.359 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi057.front.sepia.ceph.com 2023-12-02T07:24:07.359 DEBUG:teuthology.orchestra.run.smithi057:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-02T07:24:07.390 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-02T07:24:07.390 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-02T07:24:07.401 DEBUG:teuthology.orchestra.run.smithi057:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-02T07:24:07.465 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-02T07:24:07.465 DEBUG:teuthology.orchestra.run.smithi029:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-02T07:24:07.478 DEBUG:teuthology.orchestra.run.smithi057:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-02T07:24:07.640 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-02T07:24:07.654 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-02T07:24:07.655 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-02T07:24:07.686 DEBUG:teuthology.orchestra.run.smithi057:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-02T07:24:07.726 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-02T07:24:07.742 DEBUG:teuthology.orchestra.run.smithi029:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-02T07:24:07.745 DEBUG:teuthology.orchestra.run.smithi057:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-02T07:24:07.772 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = core 2023-12-02T07:24:07.808 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern = core 2023-12-02T07:24:07.861 DEBUG:teuthology.orchestra.run.smithi029:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-02T07:24:07.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-02T07:24:07.875 DEBUG:teuthology.orchestra.run.smithi057:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-02T07:24:07.915 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi057.front.sepia.ceph.com, flagging run as failed 2023-12-02T07:24:07.916 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-02T07:24:07.932 INFO:teuthology.task.internal:Transferring archived files... 2023-12-02T07:24:07.933 DEBUG:teuthology.misc:Transferring archived files from smithi029:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-01_18:01:25-rbd-main-distro-default-smithi/7474803/remote/smithi029 2023-12-02T07:24:07.933 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-02T07:24:07.973 DEBUG:teuthology.misc:Transferring archived files from smithi057:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-01_18:01:25-rbd-main-distro-default-smithi/7474803/remote/smithi057 2023-12-02T07:24:07.973 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-02T07:24:52.604 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2023-12-02T07:24:52.621 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/yuriw-2023-12-01_18:01:25-rbd-main-distro-default-smithi/7474803/remote/smithi057/coredump/1701487523.103503.core.gz: gzip compressed data, was "1701487523.103503.core", last modified: Sat Dec 2 07:24:08 2023, max compression, original size modulo 2^32 642383872 2023-12-02T07:24:52.622 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2023-12-02T07:24:55.154 INFO:teuthology.task.internal: core looks like: /tmp/tmpl2cieq6g: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'rbd-mirror --foreground --cluster cluster2 --id mirror.0', real uid: 1000, effective uid: 1000, real gid: 1301, effective gid: 1301, execfn: '/usr/bin/rbd-mirror', platform: 'x86_64' 2023-12-02T07:24:55.224 INFO:teuthology.task.internal: dump_program: rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-02T07:24:55.225 DEBUG:teuthology.orchestra.run.smithi057:> which 'rbd-mirror --foreground --cluster cluster2 --id mirror.0' 2023-12-02T07:24:55.257 INFO:teuthology.orchestra.run.smithi057.stderr:/usr/bin/which: unrecognized option '--foreground' 2023-12-02T07:24:55.258 INFO:teuthology.orchestra.run.smithi057.stderr:/usr/bin/which: unrecognized option '--cluster' 2023-12-02T07:24:55.258 INFO:teuthology.orchestra.run.smithi057.stderr:/usr/bin/which: unrecognized option '--id' 2023-12-02T07:24:55.266 INFO:teuthology.orchestra.run.smithi057.stderr:/usr/bin/which: no cluster2 in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) 2023-12-02T07:24:55.267 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/rbd-mirror 2023-12-02T07:24:55.267 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-12-02T07:24:55.268 INFO:teuthology.orchestra.run.smithi057.stderr:/usr/bin/which: no mirror.0 in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) 2023-12-02T07:24:55.268 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi057 with status 2: "which 'rbd-mirror --foreground --cluster cluster2 --id mirror.0'" 2023-12-02T07:24:55.283 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-02T07:24:55.300 INFO:teuthology.task.internal:Not uploading archives. 2023-12-02T07:24:55.301 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-02T07:24:55.344 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-02T07:24:55.344 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-02T07:24:55.352 DEBUG:teuthology.orchestra.run.smithi057:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-02T07:24:55.368 INFO:teuthology.orchestra.run.smithi029.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Dec 2 07:24 /home/ubuntu/cephtest 2023-12-02T07:24:55.368 INFO:teuthology.orchestra.run.smithi029.stdout: 262167 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Dec 2 07:24 /home/ubuntu/cephtest/archive 2023-12-02T07:24:55.368 INFO:teuthology.orchestra.run.smithi029.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 2 07:24 /home/ubuntu/cephtest/archive/syslog 2023-12-02T07:24:55.368 INFO:teuthology.orchestra.run.smithi029.stdout: 265280 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Dec 2 02:36 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-12-02T07:24:55.368 INFO:teuthology.orchestra.run.smithi029.stdout: 265245 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Dec 2 02:36 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-12-02T07:24:55.368 INFO:teuthology.orchestra.run.smithi029.stdout: 265244 104 -rw-r--r-- 1 ubuntu ubuntu 103945 Dec 2 07:24 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-02T07:24:55.368 INFO:teuthology.orchestra.run.smithi029.stdout: 402659 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 2 07:24 /home/ubuntu/cephtest/archive/audit 2023-12-02T07:24:55.369 INFO:teuthology.orchestra.run.smithi029.stdout: 402938 236 -rw------- 1 ubuntu ubuntu 238103 Dec 2 07:24 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2023-12-02T07:24:55.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-02T07:24:55.370 INFO:teuthology.orchestra.run.smithi029.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-12-02T07:24:55.370 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_bed7681480ccb0192b91a35e691658867e1e96b7/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi029 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-12-02T07:24:55.379 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-02T07:24:55.395 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-02T07:24:55.446 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: RuntimeError: error during thrashing 2023-12-02T07:24:55.447 INFO:teuthology.nuke:Checking targets against current locks 2023-12-02T07:24:55.455 INFO:teuthology.orchestra.run.smithi057.stdout: 262166 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Dec 2 07:24 /home/ubuntu/cephtest 2023-12-02T07:24:55.455 INFO:teuthology.orchestra.run.smithi057.stdout: 402710 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Dec 2 02:48 /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-02T07:24:55.455 INFO:teuthology.orchestra.run.smithi057.stdout: 402711 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Dec 2 02:49 /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2023-12-02T07:24:55.456 INFO:teuthology.orchestra.run.smithi057.stdout: 416685 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 2 02:49 /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2023-12-02T07:24:55.456 INFO:teuthology.orchestra.run.smithi057.stdout: 262167 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Dec 2 07:24 /home/ubuntu/cephtest/archive 2023-12-02T07:24:55.456 INFO:teuthology.orchestra.run.smithi057.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 2 07:24 /home/ubuntu/cephtest/archive/syslog 2023-12-02T07:24:55.456 INFO:teuthology.orchestra.run.smithi057.stdout: 262223 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Dec 2 02:36 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-12-02T07:24:55.456 INFO:teuthology.orchestra.run.smithi057.stdout: 265245 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Dec 2 02:36 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-12-02T07:24:55.456 INFO:teuthology.orchestra.run.smithi057.stdout: 265244 108 -rw-r--r-- 1 ubuntu ubuntu 106923 Dec 2 07:24 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-02T07:24:55.456 INFO:teuthology.orchestra.run.smithi057.stdout: 402708 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 2 07:24 /home/ubuntu/cephtest/archive/audit 2023-12-02T07:24:55.456 INFO:teuthology.orchestra.run.smithi057.stdout: 402712 240 -rw------- 1 ubuntu ubuntu 244224 Dec 2 07:24 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2023-12-02T07:24:55.457 INFO:teuthology.orchestra.run.smithi057.stdout: 262168 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 2 03:25 /home/ubuntu/cephtest/archive/coredump 2023-12-02T07:24:55.457 INFO:teuthology.orchestra.run.smithi057.stdout: 268021 53204 -rw------- 1 ubuntu ubuntu 642383872 Dec 2 03:25 /home/ubuntu/cephtest/archive/coredump/1701487523.103503.core 2023-12-02T07:24:55.457 INFO:teuthology.orchestra.run.smithi057.stdout: 268020 65504 -rw------- 1 ubuntu ubuntu 751144960 Dec 2 03:23 /home/ubuntu/cephtest/archive/coredump/1701487429.96602.core 2023-12-02T07:24:55.458 INFO:teuthology.orchestra.run.smithi057.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-12-02T07:24:55.476 DEBUG:teuthology.nuke:shortname: smithi029 2023-12-02T07:24:55.476 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-02T07:24:55.499 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-01_18:01:25-rbd-main-distro-default-smithi/7474803', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-02 02:25:59.240375', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHhvEccHXaj9KecRdhJ2R7k6deK/FnDjkxOTwaL94Ze0GkNEzZbozrPwNc2SMmJZq3gdDq6vTnghyy+IyRAxjQw='} 2023-12-02T07:24:55.503 DEBUG:teuthology.nuke:shortname: smithi057 2023-12-02T07:24:55.503 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-02T07:24:55.521 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi057.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-01_18:01:25-rbd-main-distro-default-smithi/7474803', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-02 02:25:59.242125', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBcE1vhLAy0ZftcWRW+Iu01zQaKILaNE+BFn/TIqb3zF05SMgiwusZ/3kJVVvXngHrx6SL2rEFn+kkOfgIxp3HY='} 2023-12-02T07:24:55.552 INFO:teuthology.orchestra.console.smithi029:Power off 2023-12-02T07:24:55.553 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-02T07:24:55.581 DEBUG:teuthology.orchestra.console.smithi029:power off output: Chassis Power Control: Down/Off 2023-12-02T07:24:55.581 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-02T07:24:55.598 INFO:teuthology.orchestra.console.smithi057:Power off 2023-12-02T07:24:55.599 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-02T07:24:55.609 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2023-12-02T07:24:55.623 DEBUG:teuthology.orchestra.console.smithi057:power off output: Chassis Power Control: Down/Off 2023-12-02T07:24:55.623 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-02T07:24:55.650 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on 2023-12-02T07:24:59.611 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-02T07:24:59.651 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-02T07:24:59.723 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2023-12-02T07:24:59.764 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on 2023-12-02T07:25:03.726 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-02T07:25:03.765 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-02T07:25:03.837 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is off 2023-12-02T07:25:03.875 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is off 2023-12-02T07:25:03.938 INFO:teuthology.orchestra.console.smithi029:Power off completed 2023-12-02T07:25:03.976 INFO:teuthology.orchestra.console.smithi057:Power off completed 2023-12-02T07:25:04.083 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} duration: 17229.490659475327 failure_reason: 'Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi057 with status 110: ''mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bed7681480ccb0192b91a35e691658867e1e96b7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\''\'' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=1533c33cce4b425c9df819b80ad0c7da status: fail success: false 2023-12-02T07:25:04.083 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-02T07:25:04.156 INFO:teuthology.run:FAIL